CN103035123B - Abnormal data acquisition methods and system in a kind of traffic track data - Google Patents

Abstract

The present invention is applicable to technical field of intelligent traffic, and provide abnormal data acquisition methods and system in a kind of traffic track data, described method comprises: the gps data obtaining traffic track to be analyzed; Determine the Minimum Area that described gps data distributes, and described Region dividing is become the subregion of multiple area equation; Add up the number of the gps data point comprised in every sub regions; Analyze the virtual route with gps data Point matching in every sub regions; According to number and the virtual route of the gps data point comprised in described every sub regions, whether the gps data obtained in each subregion is abnormal data.The present invention does not need to rely on extra database by abnormal data deterministic process, and because of the consumption of the cost and data analysis process that this reduce usage data storehouse, and implementation procedure is simple.

Description

Method and system for acquiring abnormal data in traffic trajectory data

technical field

The invention belongs to the technical field of intelligent transportation, and in particular relates to a method and system for acquiring abnormal data in traffic trajectory data.

Background technique

In recent years, the number of urban motor vehicles has increased rapidly, which has caused many problems such as traffic jams, parking difficulties, and difficulty in taking taxis, which seriously affect the quality of ordinary people's travel. At the same time, the city's traffic network is becoming more and more complex, and the requirements for a perfect management system are getting higher and higher. In the blueprint of building a smart city in the future, smart transportation is the top priority. The core part of intelligent transportation is to understand the driving rules of motor vehicles, and use this to carry out road section planning and traffic management. The realization of intelligent transportation must be based on massive data analysis of real driving data of a large number of vehicles. The popularity of vehicle-mounted GPS devices in urban traffic makes it feasible to obtain vehicle location information. A typical example is the GPS information of a taxi. All taxis will transmit the GPS data during their driving back to the server of the taxi service company to facilitate the dispatching of the headquarters, so that a large amount of GPS information can be collected throughout the urban traffic network. Research on traffic-related issues and service development based on taxi driving data has been carried out in government, business, and academia. The primary problem to be faced with the use of massive GPS data of vehicles is how to discover and deal with abnormal elements in the large amount of data. There are many reasons for the abnormal data: first, the limitation of the accuracy of civilian GPS equipment makes GPS data often scattered within a certain range of the real position; The problem gives GPS data with a large deviation; in addition, the GPS device itself may not work properly due to lack of maintenance, which is relatively common in the case of a large number of vehicles. The use of wrong GPS data will affect the results of subsequent analysis of traffic conditions, so it is of great significance to exclude abnormal elements in GPS data.

The prior art method for eliminating abnormal elements in GPS data mainly uses information other than GPS data to judge whether the GPS data is good or bad. For example, the scope information of the city can be obtained, including the boundary of the city served by the taxi, and the areas where vehicles cannot pass such as mountains, rivers, lakes, and seas, so that the GPS data that crosses the boundary or is in the restricted area can be determined as bad. In addition, the average speed of the vehicle can be calculated from the GPS data of the vehicle. If the speed value is abnormally large, for example, it is much higher than the speed limit of motor vehicles in the city, it can be considered that there is an error in the GPS data. In addition, there are many studies using road matching technology, which is a technology that matches all GPS data points to specific urban roads with the help of urban geographic information (GIS), and identifies GPS data that are difficult to match reasonably as errors data. In the above methods, the method of using the city range information is very rough, and the GPS data in the drivable area cannot be judged. The method of calculating the average vehicle speed can be used to judge part of the error data caused by multipath and the error data generated by abnormal GPS equipment, but the application range is narrow. The method of road matching using GIS not only requires accurate GIS road information, but also needs to do a lot of probability analysis when dealing with matching, which affects the performance of the method.

To sum up, the prior art method for eliminating abnormal elements in GPS data needs information other than GPS data to judge whether the GPS data is good or bad, and the judgment process is complicated.

Contents of the invention

The purpose of the embodiments of the present invention is to provide a method for acquiring abnormal data in traffic track data, which aims to solve the problem of eliminating abnormal elements in GPS data in the prior art, which needs to use information other than GPS data to analyze the abnormal elements in GPS data. Judgment is good or bad, and the judgment process is complicated.

In order to achieve the above purpose, embodiments of the present invention provide the following technical solutions:

The embodiment of the present invention is achieved in this way, a method for acquiring abnormal data in traffic trajectory data, the method comprising:

Obtain GPS data of the traffic trajectory to be analyzed;

Determine the minimum area where the GPS data is distributed, and divide the area into a plurality of equal-area sub-areas;

Count the number of GPS data points contained in each sub-region;

Analyzing virtual paths matched to GPS data points in each sub-region;

According to the number of GPS data points and the virtual path included in each sub-area, whether the GPS data in each sub-area is abnormal data is obtained.

An embodiment of the present invention also provides a system for acquiring abnormal data in traffic estimation data, the method comprising:

GPS data acquisition unit, for acquiring the GPS data of the traffic track to be analyzed;

A determination unit, configured to determine the minimum area of the GPS data distribution, and divide the area into a plurality of sub-areas with equal areas;

A statistical unit, used to count the number of GPS data points contained in each sub-area;

An analysis unit for analyzing a virtual path matched with GPS data points in each sub-area;

The abnormal data obtaining unit is used to obtain whether the GPS data in each sub-region is abnormal data according to the number of GPS data points and the virtual path contained in each sub-region.

Compared with the prior art, the embodiment of the present invention has the beneficial effects of acquiring GPS data of the traffic trajectory to be analyzed, determining the minimum area where the GPS data is distributed, and dividing the area into a plurality of sub-areas with equal areas, Count the number of GPS data points included in each sub-area, analyze the virtual path that matches the GPS data points in each sub-area, and obtain each Whether the GPS data in the sub-region is abnormal data or not, the process of judging the abnormal data does not need to rely on an additional database, so the cost of using the database and the consumption of the data analysis process are reduced, and the implementation process is simple.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

Fig. 1 is the flow chart of the realization of the abnormal data acquisition method in the traffic track data provided by Embodiment 1 of the present invention;

Fig. 2 is a flow chart of the implementation of obtaining whether the GPS data in each sub-area is abnormal data provided by Embodiment 1 of the present invention;

Fig. 3a is a schematic diagram of traffic trajectory data after removal of abnormal data by the prior art method provided by Embodiment 1 of the present invention;

Fig. 3b is a schematic diagram of the traffic track data after using the method of the present invention to remove abnormal data provided by Embodiment 1 of the present invention;

Fig. 4 is a structural diagram of a system for acquiring abnormal data in traffic trajectory data according to Embodiment 2 of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

This embodiment provides a method for acquiring abnormal data in traffic track data, the method comprising:

Obtain GPS data of the traffic trajectory to be analyzed;

Determine the minimum area where the GPS data is distributed, and divide the area into a plurality of equal-area sub-areas;

Count the number of GPS data points contained in each sub-region;

Analyzing virtual paths matched to GPS data points in each sub-region;

According to the number of GPS data points and the virtual path included in each sub-area, whether the GPS data in each sub-area is abnormal data is obtained.

An embodiment of the present invention also provides a system for acquiring abnormal data in traffic estimation data, the method comprising:

GPS data acquisition unit, for acquiring the GPS data of the traffic track to be analyzed;

A determination unit, configured to determine the minimum area of the GPS data distribution, and divide the area into a plurality of sub-areas with equal areas;

A statistical unit, used to count the number of GPS data points contained in each sub-area;

An analysis unit for analyzing a virtual path matched with GPS data points in each sub-area;

The abnormal data obtaining unit is used to obtain whether the GPS data in each sub-region is abnormal data according to the number of GPS data points and the virtual path contained in each sub-region.

The realization of the present invention is described in detail below in conjunction with specific embodiment:

Embodiment one

Fig. 1 shows the flow chart of the realization of the abnormal data acquisition method in the traffic track data provided by Embodiment 1 of the present invention, which is described in detail as follows:

In S101, the GPS data of the traffic trajectory to be analyzed is acquired;

In this embodiment, the GPS data is usually expressed in the form of (longitude, latitude), which can be used as points on a two-dimensional map drawn with latitude and longitude.

In S102, determine the minimum area of the GPS data distribution, and divide the area into a plurality of sub-areas with equal areas;

In this embodiment, S102 may specifically be implemented in the following manner:

1. Determine the minimum rectangular area where the GPS data is distributed, the vertical side of the rectangle is parallel to the longitude, and the horizontal side of the rectangle is parallel to the latitude;

2. Divide the minimum rectangular area into multiple sub-areas by using equidistant meridian lines and equidistant latitude lines, and each sub-area is a rectangle.

In this embodiment, the shape of the minimum area and the sub-area may be square, circular, or rectangular, and different shapes of the minimum area may be determined according to actual processing requirements.

In this embodiment, the area can be divided into a plurality of sub-areas by means of a grid. Preferably, an equidistant grid is used to divide out sub-areas of the same size, so that all sub-areas can be evaluated for subsequent use of a consistent judgment standard. Processing provides a basis; in addition, the size of each sub-region is arbitrarily set by an equidistant grid. Preferably, each sub-region is a rectangle whose longest side is less than 5 times the average road width, so as to ensure that each sub-region contains road tracks There is basically only one road, and the road is estimated to be approximately straight in this sub-area. Of course, it cannot be too small to lose the direction information of the road track. For example, it may be set that the length of the longest side of each sub-region does not exceed 200 meters, which is about five times the average width of urban motor vehicle roads.

An example of division is given below, but it is not limited to the situation of this example: Assume that the minimum rectangular area of the geographical area where GPS data is distributed is A, the vertical side of the rectangle is parallel to the longitude, the horizontal side of the rectangle is parallel to the latitude, and the rectangular The longitude range is (h _min , h _max ), the latitude range is (l _min , l _max ), and the minimum rectangular area A is divided into sub-area sets by the equidistant meridian family and equidistant latitude family as {a _i |i=1 ,2...NxM}, wherein, i is the unique number given to each sub-area by the system, M is the number of sub-areas divided in the meridian direction, N is the number of sub-areas divided in the latitude direction, and each sub-area is also rectangular area.

In S103, count the number of GPS data points contained in each sub-region;

In this embodiment, within the minimum rectangular area (h _min , h _max ), the latitude range is (l _min , l _max ), determine the boundaries of all sub-areas a _i , judge each GPS data point, and judge its The sub-areas, and then count the number of GPS data points contained in each sub-area.

In this embodiment, because the amount of traffic track data is huge, it is preferable to use parallel processing to judge each GPS data point in the statistical process.

Further, the Hadoop parallel data processing system can be used to analyze and organize GPS data points, wherein the numbering depends on the Map operation in Hadoop, and the number of GPS data points contained in each sub-region is obtained through the Reduce operation in Hadoop, so that Obtain a series of small files named after the number of the sub-area, each of which contains information about the GPS data contained in a sub-area, for example, the number of GPS data points, and the latitude and longitude information of each GPS data point , and other additional information, such as the time stamp of the point and the number of the vehicle that need to be added when analyzing the taxi trajectory. In S104, analyzing the virtual path matched with the GPS data points in each sub-area;

In S104, analyze the virtual route matched with the GPS data points in each sub-region.

In this embodiment, preferably, the virtual path matching with the GPS data points in each sub-area can be obtained through linear regression analysis, for example, by performing linear regression analysis on the GPS data points in each sub-area, the The slope of the virtual path that best matches the GPS data points in . If there is a real road in the sub-area, the fitting virtual path should match the road; on the contrary, if there is no real road in the sub-area, all the GPS data points in it are wrong data, and the direction of the virtual path obtained at this time random.

In this embodiment, in the analysis process, preferably, parallel processing can still be used to obtain the virtual path matched with the GPS data points in each sub-area.

Further, the Hadoop framework mentioned in S103 can still be used. Specifically, the regression analysis process can be placed in the Reduce stage to obtain the virtual path matched by the GPS data points in the current sub-region.

In S105, according to the number of GPS data points and the virtual path included in each sub-area, whether the GPS data in each sub-area is abnormal data is obtained.

In this embodiment, S105 can specifically be implemented in the following manner. The specific implementation idea is: firstly, each sub-region must contain enough data points before it can continue to be analyzed. If the sub-region contains a lot of data points, then the The reliability of the sub-region is very high, and it can be directly judged that the sub-region contains correct information; on the contrary, if the number of data points contained in the sub-region is very small, it can be directly judged that the GPS data contained in the sub-region is abnormal data, please Referring to Fig. 2, it shows a flow chart of the realization of obtaining whether the GPS data in each sub-area provided by Embodiment 1 of the present invention is abnormal data, which is described in detail as follows:

In S201, it is judged whether the number of GPS data points in the current sub-region is greater than the first threshold, if not, execute S205, and if yes, execute S202.

In S202, it is judged whether the number of GPS data points in the current sub-area is greater than a second threshold, wherein the first threshold is smaller than the second threshold, if not, execute S203, and if yes, execute S204.

In S203, it is judged whether the standard deviation of the vertical distance between all GPS data points and the virtual path in the current sub-region is greater than the standard deviation threshold, if not, then perform S204, if so, then perform S205;

In this embodiment, the calculation process of the standard deviation of the vertical distance between all GPS data points and the virtual path in the current sub-region is as follows: calculate the vertical distance between all GPS data points and the virtual path, and the distance between all GPS data points and the virtual path The average value of the vertical distance of the path, according to the vertical distance of all GPS data points and the virtual path and the average value of the vertical distance of all GPS data points and the virtual path to calculate the vertical distance of all GPS data points and the virtual path standard deviation of .

In this embodiment, if all GPS data points are perfectly distributed on the real road, the virtual path obtained by regression analysis is equal to the real path, and the vertical distance between all GPS data points and the virtual path should be less than half of the road width, so The resulting standard deviation should also be smaller than the standard deviation threshold. If the standard deviation is large, it indicates that there is a high probability that the subregion does not have a true path.

In S204, it is determined that all GPS data points in the current sub-area are normal GPS data points.

In S205, it is determined that all GPS data points in the current sub-area are abnormal GPS data points;

The above three thresholds: the first threshold, the second threshold and the standard deviation threshold are determined through the statistical analysis of the relevant information of all sub-regions. The specific analysis methods that can be used include the interval ratio method, that is, the The number of GPS data points satisfies a given ratio.

The result that the data is processed by the abnormal data acquisition method provided by the present embodiment below is that the traffic trajectory data is processed: the data comes from the operating data of 20,000 taxis in one month in August 2010 in Shenzhen City, and each taxi is Send a message to the data processing center every minute, and the message format is (latitude, longitude, vehicle status, time). The storage format of the data is an ordinary text file. Each file stores the time-ordered GPS information of a taxi. All data contains 20,000 text files, with a total size of about 50G. It can be seen that by using the method of the present invention, in conjunction with the Hadoop distributed file processing system, all data can be completed in no more than 20 minutes for error detection and elimination (depending on the server performance for storing and analyzing data), which can be obtained more than traditional method for better results. As shown below, Fig. 3a is a schematic diagram of the traffic trajectory data after the abnormal data is removed by the method of the prior art; Fig. 3b is a schematic diagram of the traffic trajectory data after the abnormal data is removed by the method of the present invention. It can be seen that the data points accounting for about 20% of the total using the method of the present invention have been excluded, and most of the data points distributed outside the reasonable roads have been removed.

In this embodiment, the GPS data of the traffic trajectory to be analyzed is obtained, the minimum area of the GPS data distribution is determined, and the area is divided into a plurality of sub-areas with equal areas, and the GPS data points contained in each sub-area are counted number, analyze the virtual path matched with the GPS data points in each sub-area, and obtain whether the GPS data in each sub-area is abnormal data according to the number of GPS data points contained in each sub-area and the virtual path , the abnormal data judging process does not need to rely on an additional database, thus reducing the cost of using the database and the consumption of the data analysis process, and the implementation process is simple.

In addition, the corresponding technical means adopted in the judgment process in the embodiment of the present invention can also bring the following beneficial effects:

1. By dividing into multiple subdivided sub-regions, the coupling degree between sub-regions is reduced, so that the analysis process of massive data can be completed in parallel, and the GPS data of each sub-region is processed by parallel processing, which effectively improves The process of judging abnormal data.

2. Using the urban road width as the scale standard to divide the urban area into a rectangular grid can ensure that the road trajectory contained in each rectangular sub-area is basically unique and straight.

3. The comprehensive judgment method combining the cumulative number of GPS data points in the sub-region and the statistical analysis of the virtual path trajectory can effectively improve the accuracy of abnormal data judgment.

Embodiment two

Fig. 4 shows the structural diagram of the abnormal data acquisition system in the traffic trajectory data provided by the second embodiment of the present invention. For the convenience of explanation, only the parts related to the embodiment of the present invention are shown, and the system can be built into the traffic estimation process A software unit, a hardware unit, or a combination of software and hardware in a device.

The method includes: a GPS data acquisition unit 41 , a determination unit 42 , a statistics unit 43 , an analysis unit 44 and an abnormal data acquisition unit 45 .

GPS data acquiring unit 41, for acquiring the GPS data of the traffic trajectory to be analyzed;

A determination unit 42, configured to determine the minimum area of the GPS data distribution, and divide the area into a plurality of sub-areas with equal areas;

Statistical unit 43, used to count the number of GPS data points contained in each sub-region;

An analysis unit 44, configured to analyze a virtual path matched with GPS data points in each sub-area;

The abnormal data obtaining unit 45 is configured to obtain whether the GPS data in each sub-region is abnormal data according to the number of GPS data points contained in each sub-region and the virtual path.

Optionally, the determining unit 42 includes:

A determining module, configured to determine the smallest rectangular area of the GPS data distribution, the vertical side of the rectangle is parallel to the longitude, and the horizontal side of the rectangle is parallel to the latitude;

A division module, configured to divide the minimum rectangular area into a plurality of sub-areas by means of equidistant meridian groups and equidistant parallel groups, and each sub-region is a rectangle.

Optionally, the longest side of each sub-region is a rectangle whose length is less than twice the average road width.

Optionally, the analysis unit 44 is specifically configured to obtain a virtual route matching with GPS data points in each sub-area through linear regression analysis.

Optionally, the abnormal data acquisition unit 45 is specifically configured to determine whether the number of GPS data points in the current sub-area is greater than a first threshold;

If it is less than or equal to the first threshold, all GPS data points in the current sub-area are abnormal GPS data points;

If greater than the first threshold, when judging whether the number of GPS data points in the current sub-area is greater than the second threshold;

If it is less than or equal to the second threshold, it is judged whether the standard deviation of the vertical distance between all GPS data points in the current sub-region and the virtual path is greater than the standard deviation threshold, if it is less than or equal to the standard deviation threshold, then all GPS data points in the current sub-region are normal If the GPS data point is greater than the standard deviation threshold, all GPS data points in the current sub-area are abnormal GPS data points, wherein the first threshold is less than the second threshold;

If it is greater than the second threshold, all GPS data points in the current sub-area are normal GPS data points.

The abnormal data acquisition system in the traffic trajectory data provided by the embodiment of the present invention can be used in the first corresponding method embodiment above. For details, refer to the description of the first embodiment above, and will not be repeated here.

It is worth noting that in the above system embodiments, the units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit It is only for the convenience of distinguishing each other, and is not used to limit the protection scope of the present invention.

In addition, those of ordinary skill in the art can understand that all or part of the steps in the methods of the above-mentioned embodiments can be completed by instructing related hardware through programs, and the corresponding programs can be stored in a computer-readable storage medium. Storage media, such as ROM/RAM, magnetic disk or optical disk, etc.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (6)

1. a method for acquiring abnormal data in traffic track data, characterized in that, the method comprises: Obtain GPS data of the traffic trajectory to be analyzed; Determine the minimum area where the GPS data is distributed, and divide the area into a plurality of equal-area sub-areas; Count the number of GPS data points contained in each sub-region; Analyzing virtual paths matched to GPS data points in each sub-region; According to the number of GPS data points and the virtual path contained in each sub-area, whether the GPS data in each sub-area is abnormal data is obtained; The virtual path that the analysis matches with the GPS data points in each sub-area is specifically: Obtain a virtual path matching the GPS data points in each sub-area through linear regression analysis; According to the number of GPS data points and the virtual path contained in each sub-area, obtaining whether the GPS data in each sub-area is abnormal data is specifically: Judging whether the number of GPS data points in the current sub-area is greater than the first threshold; If it is less than or equal to the first threshold, all GPS data points in the current sub-area are abnormal GPS data points; If greater than the first threshold, when judging whether the number of GPS data points in the current sub-area is greater than the second threshold; If it is less than or equal to the second threshold, it is judged whether the standard deviation of the vertical distance between all GPS data points in the current sub-region and the virtual path is greater than the standard deviation threshold, if it is less than or equal to the standard deviation threshold, then all GPS data points in the current sub-region are normal If the GPS data point is greater than the standard deviation threshold, all GPS data points in the current sub-area are abnormal GPS data points, wherein the first threshold is less than the second threshold; If it is greater than the second threshold, all GPS data points in the current sub-area are normal GPS data points. 2. The method according to claim 1, wherein the determining the minimum area of the GPS data distribution, and dividing the area into a plurality of equal sub-areas is specifically: Determine the minimum rectangular area of the GPS data distribution, the vertical side of the rectangle is parallel to the longitude, and the horizontal side of the rectangle is parallel to the latitude; The smallest rectangular area is divided into a plurality of sub-areas by means of equidistant meridian groups and equidistant parallel groups, and each sub-region is a rectangle. 3. The method according to claim 2, wherein the length of the longest side of each sub-region is a rectangle less than twice the average road width. 4. A system for acquiring abnormal data in traffic estimation data, characterized in that the system comprises: GPS data acquisition unit, for acquiring the GPS data of the traffic track to be analyzed; A determination unit, configured to determine the minimum area of the GPS data distribution, and divide the area into a plurality of sub-areas with equal areas; A statistical unit, used to count the number of GPS data points contained in each sub-area; An analysis unit for analyzing a virtual path matched with GPS data points in each sub-area; An abnormal data acquisition unit, used to obtain whether the GPS data in each sub-area is abnormal data according to the number of GPS data points contained in each sub-area and the virtual path; The analysis unit is specifically used to obtain a virtual path matched with GPS data points in each sub-area through linear regression analysis; The abnormal data acquisition unit is specifically used to determine whether the number of GPS data points in the current sub-area is greater than the first threshold; If it is less than or equal to the first threshold, all GPS data points in the current sub-area are abnormal GPS data points; If greater than the first threshold, when judging whether the number of GPS data points in the current sub-area is greater than the second threshold; If it is less than or equal to the second threshold, it is judged whether the standard deviation of the vertical distance between all GPS data points in the current sub-region and the virtual path is greater than the standard deviation threshold, if it is less than or equal to the standard deviation threshold, then all GPS data points in the current sub-region are normal If the GPS data point is greater than the standard deviation threshold, all GPS data points in the current sub-area are abnormal GPS data points, wherein the first threshold is less than the second threshold; If it is greater than the second threshold, all GPS data points in the current sub-area are normal GPS data points. 5. The system according to claim 4, wherein the determining unit comprises: A determining module, configured to determine the smallest rectangular area of the GPS data distribution, the vertical side of the rectangle is parallel to the longitude, and the horizontal side of the rectangle is parallel to the latitude; A division module, configured to divide the minimum rectangular area into a plurality of sub-areas by means of equidistant meridian groups and equidistant latitude groups, and each sub-region is a rectangle. 6. The system according to claim 4, wherein each sub-area is a rectangle whose longest side is less than 5 times the average road width.