Background & Summary

Accurate, timely, and representative in-situ observations across large areas have always been needed to report statistics on land use, land cover, and the environment. Precise geo-located in-situ information is also indispensable to train and validate algorithms that characterize the Earth’s surface based on remotely sensed observations. Comprehensive and thematically rich in-situ data can lead to better classifiers and more accurate multi-temporal land surface mapping. This is especially true since increasingly frequent and detailed Earth Observations are being made, for instance by the fleet of Sentinel satellites of the EU’s Copernicus program. These developments are opening avenues to better combine classical statistical surveying and Earth Observation (EO) derived products in the domains of land use and land cover change and environmental monitoring (e.g1.).

Historical background

The Land Use/Cover Area frame Survey (LUCAS) in the European Union (EU) was set-up exactly to provide such statistical information2. It represents a triennial in-situ land cover and land use data collection exercise that extends over the whole of the EU’s territory (https://ec.europa.eu/eurostat/web/lucas). The LUCAS project was implemented following Decision 1445/2000/EC of the European Parliament and of the Council of 22 May 2000 “On the application of area-frame survey and remote-sensing techniques to the agricultural statistics for 1999 to 2003” and has continued since. While the LUCAS survey concept was initiated and tested in 2001 and 20033, it has been restructured in 20064 and then slightly modified to result in the actual survey design5. In 2006, Eurostat, the statistical office of the EU, launched a pilot survey project in 11 countries to test the stratified sampling design. The primary focus was on agricultural areas with emphasis given to easily accessible points. Since then Eurostat has carried out LUCAS surveys every three years with the survey design ever evolving, however the LUCAS core component (i.e. the identification of the point, and the surveying of specific variables on different aspects of land cover, land use, and land and water management6), has remained comparable for all five surveys.

Survey design summary

LUCAS collects information on land cover and land use, agro-environmental variables, soil, and grassland. The surveys also provide spatial information to analyse the mutual influences between agriculture, environment, and countryside, such as irrigation and land management. The in-situ point data is collected according to a harmonised classification with separate land cover and land use codes. Data quality is assured by a regular two-level quality control (i.e. internal and external), in which all points are evaluated by quality controllers (see7 for more details). At each LUCAS point, standard variables are collected including land cover, land use, environmental parameters (the so called micro data), as well as one downward facing photo of the point (P) and four landscape photos in the cardinal compass directions (N, E, S, W). Additionally to the core variables collected, other specific modules were carried out on demand such as (i) the transect of 250 m to assess transitions of land cover and existing linear features (2009, 2012, 2015), (ii) the topsoil module (2009, 2012 (partly), 2015 and 2018), (iii) the grassland module (2018), and (iv) the Copernicus module collecting the homogeneous and continuous extent of land cover on a 50-m radius (2018)8. Due to the specificity of these modules, their corresponding collected data are not included in the data harmonisation presented in this paper. The topsoil module datasets for 2009, 2012 and 2015 were harmonised and documented separately as an open-access dataset of topsoil properties in the EU9.

LUCAS is a two phase sample survey. The LUCAS first phase sample is a systematic selection of points on a grid with a 2 km spacing in Eastings and Northings covering the whole of the EU’s territory10. Currently, it includes around 1.1 million points (Fig. 1) and is referred to as the master sample. Each point of the first phase sample is classified in one of ten land-cover classes via visual interpretation of ortho-photos or satellite images11. The core sampling and survey methods have remained the same throughout the five surveys. Nevertheless evolving goals of the surveys have led to slightly different sample point allocations for different land covers. In 2006, the main objective was to “make early estimates of the main crop areas”, along with the ability to collect information on agri-environmental indicators in the context of the monitoring of the Common Agriculture Policy (CAP)3. In 2009, the main objective was to estimate areas, especially in conjunction with other data sources such as Corine Land Cover (CLC)10. In 2018, the main objective was to “monitor social and economic use of land as well as ecosystems and biodiversity”5. Additionally, in 2018, a linear logistic regression model based on LUCAS 2015 and 16 additional variables were used as co-variates to forecast the most probable land cover for each of these points5. From this stratified first phase sample, the second phase sample of points is selected to obtain the desired statistically representative spatial distribution of sampled land cover classes according to the first phase visual interpretation. With LUCAS 2018 this amounts to 337845 points, out of which approximately 240000 points are visited in the field by surveyors to collect additional information that cannot be assessed remotely.

Fig. 1
figure 1

Schematic overview of the LUCAS and harmonisation methodologies. The left side illustrates the sampling at the basis of the production of the LUCAS primary data. The top right side shows the raw base data (micro data). The process of harmonising is contained within the multi-year harmonised aggregation block and is the subject of the following two sections. The bottom right presents the four main outputs associated with this manuscript (more in section Data Records) - a harmonised, legend-explicit, multi-year, ready-to-use, version of the LUCAS micro data (section Overview of multi-year harmonised LUCAS survey database49,), a database with all cardinal-direction landscape and point photos collected during the surveys, including their respective EXIF attributes (section Overview of EXIF photos database, EXIF table49, photos on https://gisco-services.ec.europa.eu/lucas/photos/, the survey geometries49 and a R package to generate the data51.

Full size image

The in-situ nature of the survey implies that the majority of the data are gathered through direct observations made by surveyors on the ground. Those points which are unlikely to change and points which are too difficult to access are classified by photo-interpretation in the office, using the latest available ortho-photos or Very High Resolution (VHR) images. Although most of the points a-priori assigned for in-situ assessment can effectively be visited in the field, those that cannot be reached, because of lack of access to the point or the point location being at more than 30 minutes walking distance from the closest point reachable by car. Those points are thus photo-interpreted on ortho-photos or Very High Resolution (VHR) images in the field by the field surveyor. Furthermore, sometimes a significant difference exists between the theoretical LUCAS point and the actual GPS location reached by the surveyor. Observations are collected for the LUCAS point, while the photos are taken at the actual GPS location. Both locations and the distance between them is noted down.

Previous LUCAS use cases and shortcomings

In the scientific literature, LUCAS land cover and land use survey data have been used to derive statistical estimates2, to describe land cover/use diversity at regional level12, and its sampling frame was used as a basis for various applications including assessing the availability of crowd-sourced photos potentially relevant for crop monitoring across the EU13. LUCAS was designed to derive statistics for area estimation (e.g3. and10). Recently, several researchers have started to use LUCAS data in large scale land cover mapping processes, especially as a source of training and/or validation data for supervised classification approaches at regional/national scale14,15,16,17,18,19,20.

Several drawbacks become apparent when working with the original LUCAS datasets. While the inconsistencies could be due to the enumerators’ subjectivity in interpretation of the legends and the legend itself, it is also related to the complexity of the field survey: large number of surveyors (>700), complex documentation for the enumerators (>400 pages combining all the documents), translated to 20 languages. These drawbacks hinder the further use of the LUCAS data by the scientific community as a whole and in particular by users who are active in emerging fields of big data analytics, data fusion, and computer vision. Such drawbacks include:

  • Inconsistencies and errors between legends and labels from one LUCAS survey to the next which is hampering temporal analysis.

  • Missing internal cross-references in the datasets that would facilitate computation and linking observed variables, photos, etc.

  • The original full resolution photos taken at each surveyed point are not available for download.

  • The lack of a single-entry point or consolidated database hampering automated processing and big data analysis.

Therefore, we have gone through an extensive process of cleaning by semantic and topological harmonisation, along with connecting the originally disjoint LUCAS datasets in one consolidated database with hard-coded links to the full-resolution photos, openly accessible along with this paper.

Methods

Having contextualized the LUCAS survey, we proceed with describing the full methodological workflow to harmonise the data, as schematically shown in Fig. 1. The Sampling and Survey sub-figures provide an overview of the methodological framework of the LUCAS data collection itself (see previous section Background & Summary). The Data aggregation and Results sub-figures illustrate the work carried out in this study. The datasets collected during the five surveys (in 2006, 2009, 2012, 2015, 2018) are the main LUCAS products available (more in section Micro data collection and documentation (Protocol 1)). These datasets and their respective data documentation were used to create the multi-year harmonised database. The harmonisation process is described below and in Table 1. Associated with the summary Table 1, the Table 2 provides name changes, the Table 3 provides the new columns added, the Online-only Table 1 provides the missing column adding and the Online-only Table 2 provides the variable re-coding. The results are consolidated in one single consistent and legend-explicit table along with hard-coded links to the full resolution photos (stored on the GISCO, https://gisco-services.ec.europa.eu/lucas/photos/). The LUCAS primary data includes alpha-numerical variables and field photographs linked to the geo-referenced points.

Table 1 Aggregation of micro data - summarizing the different steps applied to harmonise the survey data.
Full size table
Table 2 Table of renamed variables.
Full size table
Table 3 Table of newly added columns.
Full size table

Micro data collection and documentation (Protocol 1)

The first step is to collect the data from the source for each survey year (see Table 1 Source). The raw micro data for the harmonised database presented here are the five LUCAS campaigns, which can be downloaded from the official Eurostat website (https://ec.europa.eu/eurostat/web/lucas). The LUCAS micro data for 200621 is divided into a separate file for each of the 11 surveyed countries (Belgium22, Czechia23, Germany24, Spain25, France26, Italy27, Luxembourg28, Hungary29, Netherlands30, Poland31, and Slovakia32). The LUCAS micro data is provided aggregated for all countries for every other survey years, whereby the data can also be downloaded separately by country (200933, 201234, 201535) in CSV format and 201836 in 7z zipped format. The second step is to collect the documentation that facilitates translating the alpha-numerical class-description in the original datasets into explicit information. For 2006, 2009 and 2012, the survey data comes with a content descriptor (200637, 200938, 201239), though the content descriptor doesn’t necessarily have the same number of variables as the data; and the variables themselves sometimes have a slightly different name. These inconsistencies were resolved with assistance from the technical documents (LC1 (Instructions, 200640, 200941, 201242) and LC3 (Classification, 200643, 200944, 201245). From 2015 and 2018, the data is provided with a record descriptor (201546, 201847), which contains information on variable name, data type and description in a more consolidated fashion, making it easier to find information about the relevant variable.

The third and final step in Protocol 1 is the generation of the mapping files used for value recoding. The workflow maps the ascertained relationship between those variables that are the same but have changed in name or alpha-coding between surveys. To recode all variables coherently from one survey to the next, the original data is changed permanently. All transformations are done by recoding ordinal variables to be compliant with the encoding of variables used in the last survey (2018). These mappings serve as a blueprint for the transformation and data integration described in Protocol 2.

Micro data harmonisation (Protocol 2)

The harmonisation workflow, alongside the performed database consistency checks, is shown in Fig. 2 and the code is described in code section (section Code availability). The general principle of the harmonisation workflow was to convert all the field legends to fit with the latest i.e. the 2018 database layout (the next LUCAS is planned for 2022).

Fig. 2
figure 2

Processing workflow to harmonise the survey data. Asterix used to indicate steps after which there is a performed consistency check (Merge into single table, Align mapping CSVs, and Convert encoding to label).

Full size image

Some notable changes in the source tables had to be made in order to make the harmonisation and subsequent merger into one complete table possible. This was accomplished with the above-mentioned instance-mapping files (Section Micro data collection and documentation (Protocol 1)). All manipulations executed over the separate tables prior to the merger are listed in Table 1 under heading ‘Protocol 2’:

  1. 1.

    Rename columns - iteratively renaming columns to align them with the last (in this case 2018) survey. Performed on all tables but 2018 by using the Rename_cols() function from the package.

  2. 2.

    Add photo column 2006 - adds columns photo_north, photo_south, photo_east, photo_west, and photo_point on account of them missing from the 2006 base data. Adding is done by cross-referencing the EXIF picture database (see section Overview of EXIF photos database). Performed solely on table for 2006 by using the Add_photo_field_2006() function.

  3. 3.

    Add Missing columns - iteratively adding all columns that are present in one table and not present in the others. Performed on all tables by using the Add_missing_cols() function.

  4. 4.

    Add new columns - iteratively adding all newly created columns. These include the variables ‘letter group’, ‘year’, and ‘file_path_gisco_n/s/e/w/p’ (for more information check Online-only Table 3). Performed on all tables using the Add_new_cols() function.

  5. 5.

    Upper case - iteratively converting all characters of selected fields to upper case. Performed on all tables using the Upper_case() function.

  6. 6.

    Re-code variable - iteratively re-coding selected variables according to created mapping CSV files, designed referring back to the reference documents. Performed on all tables but 2018 by using the Recode_vars() function.

  7. 7.

    Order columns - iteratively ordering all columns according to the template from the 2018 survey. Performed on all tables but 2018 by using the Order_cols() function.

Merge and post-processing (Protocol 3)

The third part of the harmonisation process includes the merging of the harmonised tables of each survey year plus additional steps listed below before exporting the final data outputs.

  1. 1.

    Merge into single table - Merge the five harmonised tables to one unique table via Merge_harmo() function. Consistency check performed after this successful execution on newly generated Table.

  2. 2.

    Correct theoretical location - Applying a correction of the values of columns th_long and th_lat for merged harmonised table according to the latest LUCAS grid via the Correct_th_loc() function.

  3. 3.

    Add geometry columns - Location of theoretical point(th_geom), location of lucas survey (gps_geom), lucas transect geometries (trans_geom) and distance between theoretical and survey point (th_gps_dist). Done by the Add_geom() function.

  4. 4.

    Create database tags - Primary key, index, and spatial index via the Create_tags() function.

  5. 5.

    Add number of visits column - column to show the number of times between the years when the point was visited thanks to the Add_num_visits() function.

  6. 6.

    Align mapping CSVs - Corrects any typo, spelling mistake, or spelling difference in the user-created mapping CSVs, used to generate labels in subsequent function that converts encoding to label by aligning them to the mapping CSV of the latest survey. Done by the Align_map_CSVs() function. Consistency check performed after this successful execution on newly generated mapping CSVs.

  7. 7.

    Convert encoding to label - Create columns with labels for coded variables and decodes all variables where possible to explicit labels. Performed with User_friendly() function. Consistency check performed after this successful execution.

  8. 8.

    Final column order - Re-order columns of final tables with the Final_order_cols() function.

  9. 9.

    Remove variables - optional function to remove variables which the technician deems not necessary for the new harmonised product. Done with the Remove_vars() function.

  10. 10.

    Update record descriptor - Updates Record descriptor by adding a field (year) showing the year for which the variable exists and removing variables listed in the optional function for removing variables from record descriptor. Done with the Update_RD() function.

The workflow ends with the output exports. The table is exported as CSV and the geometries as shapefiles. The full workflow is dependent on two software prerequisites. Firstly, one must have a running PostgreSQL server, and secondly, an installation of R (more about the versions used in section Code availability). The pipeline is provided as a R package for ease of reproducibility and transparency (section Code availability).

Full resolution LUCAS photos

In addition to the alphanumerical and geometry information of the survey, a complete database with full-resolution point and landscape photos was set up with photos retrieved from Eurostat. This archive was organised as a table with all the exchangeable image file (EXIF) variables for each of the images, among which a unique file path, as stored on the Eurostat GISCO server for easy retrieval by other researchers. Besides the EXIF attributes, each photo is also hard-coded with the respective point ID of the LUCAS point and the year of survey. The photos’ metadata were extracted with ExifTool (v 10.8)48 resulting in a database of photos that was compared for completeness with the survey data records. The hard-coded HTTPS links to each photo in the consolidated database allow for large data volume queries and selection tasks.

Data Records

The first section Storage describes each data-set provided along with this manuscript including the table, photo, and geometry databases along with the R package created to compile and construct all the data. The second section Overview of multi-year harmonised LUCAS survey database provides an outline of the resulting harmonised database and the last section Overview of EXIF photos database provides an overview of the photo database.

Storage

  1. 1.

    Multi-year harmonised LUCAS survey data. The harmonised database (available for download here49 and also archived as compressed folder here50) contains 106 variables and 1351293 records corresponding to a unique combination of survey year and field location. The same dataset is also available for each year with a different file for users interested only in one specific survey. The database is provided with a Record descriptor (Online-only Table 3 presents a summary, the complete table is available here in CSV format49 in the supporting files). This record descriptor specifies variable name, data type, range of possible values and meanings. In the documentation one can find more information about the variable and a short description, along with comments concerning the variable that the authors have deemed important. Additionally, the tables in LUCAS-Variable and Classification Changes, in the supporting files, contain documentation for users to quickly identify the differences between LUCAS campaigns and the harmonised database. The file contains four tables:

  1. 1.

    “References”: Description and a legend of the used colors of the different tables;

  2. 2.

    “Harmonised DB”: a comparison of all the collected variables of the 2018 survey with the variables of the harmonised database and an overview of the actions to harmonise the data;

  3. 3.

    “Variable changes”: an overview/ comparison of all collected variables between all campaigns from 2009 to 2018 highlighting the changes;

  4. 4.

    “LC (LU) changes”: an overview of the possible LC and LU codes of each campaign highlighting the changes.

  1. 2.

    LUCAS survey geometries/point locations. To facilitate spatial analysis and usability, three types of geometries are provided as distinct shapefiles (see the geometries folder downloadable on49):

  1. 1.

    LUCAS theoretical points (th_long, th_lat),

  2. 2.

    LUCAS observed points (gps_lon, gps_lat) and

  3. 3.

    LUCAS transect lines (250-m east looking lines).

  1. 3.

    High resolution LUCAS photo archive. The 5.4 millions of photos collected during the five surveys are available at https://gisco-services.ec.europa.eu/lucas/photos/. For each in-situ point, landscape (N, E, S, W), and point (P) photos are available. The EXIF information of all the photos were extracted and are provided as an additional table (lucas_harmo_exif.csv49).

  2. 4.

    R package. The scripts to harmonise the LUCAS data is provided as an open source R package along with the documentation51.

Overview of multi-year harmonised LUCAS survey database

Among the data provided with the current study described in the previous section, the multi-year harmonised LUCAS survey database contains the five LUCAS surveys, i.e. a total of 1351293 observations that have been made at 651676 unique locations (Table 4). The total number of surveyed points has increased significantly from the 2006 pilot study (168401) to 2015 (340143) (Table 4). This rise is mainly due to the increase in terms of thematic richness, scope, and scale of the study from what was primarily an evaluation of agricultural areas (2006) to a more holistic and exhaustive inspection of the EU territory. Further, the total number of surveyed countries increased from 11 in 2006 to 28 in 2018 (Table 4). Over the five surveys, 1 031 813 observations (76.36%) were done in-situ. Out of these in-situ observations, 94% have been surveyed within 100 m distance of the theoretical LUCAS point and 6% were more than 100 m away from the point. The proportion of points where actual in-situ data was collected has decreased from 92.18% in 2006 to 63.67% in 2018. Furthermore, 10.92% of the points (i.e. 147574) that were visited in-situ turned out not be accessible in practice and are photo-interpreted in the field. The number of points surveyed per country and per year ranged between 79 (Malta) to 48215 (France). Finally, over the five surveys, 1677 points were out of national territory, i.e. “NOT EU” corresponding to water outside national borders or countries including Russia, Turkey, Albania and Switzerland).

Table 4 Number of LUCAS points per country and per year.
Full size table

Figure 3 provides the accumulative frequency of assigned level-3 classes (out of 77 classes in total) to the surveyed points, sorted by reference year. Land Cover/Land Use (LC/LU) classification specifications can be found in the new reference document, containing the harmonised C3 legend (see Harmonized C3 legend in49).

Fig. 3
figure 3

Distribution of land cover classes in the multi-year harmonised LUCAS database. In cases where survey years are not present please orientate oneself with reference to adjacent classes of the same color. Counting for the distribution of each class begins at 2018 and ends with 2006 due to the relative abundance of 2018 in terms of classes compared to other years.

Full size image

The classification system follows rules on spatial and temporal consistency - it can be applied and compared both between locations in the EU and by survey years. Additionally, excluding 2006, it is ‘as much as possible’ compatible with other existing LC/LU systems (e.g. Food and Agriculture Organization (FAO), statistical classification of economic activities in the European Community (NACE) (2009–2018) and fulfills the specifications of the European Infrastructure for Spatial Information in Europe (INSPIRE) (2015-2018)). To inform about changes in two consecutive surveys, the data providers describe the adjustments to the terminology in the documentation. The 3-level legend system is arranged hierarchically, whereby the first level (letter group) corresponds to the eight main classes obtained by ortho-photo-interpretation during the second level stratification phase (Fig. 1); the second and third level, representing subcategories of these main classes are indicated by a combination of the letter group and further digits.

The number of point visits is shown in Table 5. Some LUCAS points were visited once in 15 years (n = 332605) while others were visited each time, thus totaling five visits (n = 35204). This means that 651780 locations were at least visited once. Figure 4 shows a map with the visit frequency for each point over Europe.

Table 5 Number of LUCAS points and visits.
Full size table
Fig. 4
figure 4

Number of visits to each LUCAS survey point over the five surveys between 2006 and 2018, 651780 points were at least surveyed once. Visit ranges from one to five.

Full size image

Overview of EXIF photos database

The available photos (N, E, S, W, P, i.e. North, East, South, West, and Point) were catalogued totaling 5440459 photos for the 5 surveys (see Table 6 for detailed distribution). The lucas_harmo_exif.csv table contains the essential and available LUCAS EXIF information (27 variables) for all the photos 2006, 2009, 2012, 2015, 2018. While the observation location is recorded by the surveyor during the LUCAS field survey (gps_lon, gps_lat), the digital cameras with GPS could also capture the location where the photos were taken as well as the orientation, i.e. the azimuth angle. In the first surveys, the digital camera and the GPS were separate devices. The orientation was determined with a traditional compass. The data were used to cross-validate the geo-location reported during the survey. To assess the availability of this information, the EXIF information of the 5440459 photos was retrieved. As summarised in the two last columns of Table 6, the photos with geo-location information have increased considerably through time, i.e. 0% in 2006, 5.4% in 2009, 34.2% in 2012, 68.5% in 2015 and finally 72.9% in 2018. For azimuth angle, there is no information on orientation for the photos taken in 2006 and 2009. However, respectively 15.3%, 22%, and 6.7% of the photos have EXIF orientation information for 2012, 2015, and 2018.

Table 6 Number of LUCAS photos per year, per type (N, E, S, W, P) with proportions that have EXIF geo-location (Location [%]) and orientation information (Orientation [%]).
Full size table

Each point surveyed has potentially five photos (N, E, S, W, P) per surveyed year (Fig. 5(a)). The EXIF table database is a table of records, corresponding to the photos taken in the cardinal orientations plus the point for each one of the points for the five surveys. The table holds information on the point ID, year of survey, path to the full resolution image and an wide variety of EXIF attributes, including coordinates, orientation, camera model, exact time and date, Eurostat metadata, etc.

Fig. 5
figure 5

Overview of the data available for a LUCAS point that was visited five times: (a) Point, North, East, South and West photos for 2006, 2009, 2012, 2015 and 2018, (b) Location of the point in the EU, (c) Zoom showing the point (3-m diameter in green, 50-m diameter in dashed red), (d) Visit frequency on a 20 by 20 km square centered on the point, and (e) In-situ land cover observation of the point for the different years.

Full size image

It was decided that having this information in a separate table is more sensible in terms of storage size and accessibility, whereby cross-table checks can easily be performed by executing joins between the tables based on point ID and year of survey. By combining this information from the two tables (i.e. the multi-year harmonised LUCAS survey database and the EXIF table database) one arrives at a significantly large set of labeled examples, corresponding to images of the 77 different types of recorded land cover.The background RGB imagery for (c) and (d) is obtained from “Map data ©2019 Google”.

Technical Validation

The first part of this section briefly summarises the LUCAS field surveys quality check. The section then focuses on analyses carried out specifically to support the technical quality of the multi-year harmonised LUCAS database process.

The LUCAS surveyed observations are subject to detailed quality checks (see LUCAS metadata52 and the data quality control documents available for 200953, 201254, 201555). First, an automated quality check verifies the completeness and consistency after field collection. Second, all surveyed points are checked visually at the offices responsible for collection. Third, an independent quality controller interactively checks 33% of the points for accuracy and compliance against pre-defined quality requirements, including the first 20% observations for each surveyor, to prevent systematic errors during the early collection phase.

The presented data consolidation effort seeks to enhance the quality of an existing product. Ensuring data quality by harmonisation throughout the years is thus essential. Data quality was ensured by taking into account validity, accuracy, completeness, consistency, and uniformity throughout data processing (Fig. 2):

  • Validity of the harmonised database was ensured via data type (for which information can be found in the record descriptor) and a unique constraint of a composite key (consisting of the point ID and year of survey).

  • Accuracy of the data relies on the source data for which the quality was assessed as described in the previous paragraphs.

  • Completeness checking shows that since several variables have been added over the years, many missing values exist. In such cases, fields were populated with null values. Consistency across surveys has been enhanced. All surveys were harmonised towards the 2018 survey.

  • Consistency of the presented dataset was internally ensured through running checks at various stages of processing.

  • Uniformity checks revealed that the geographical coordinates in columns th_long and th_lat show different locations between some survey years. In the interest of complete uniformity, it was decided to have the values of these variables hard coded from the LUCAS grid. Because the LUCAS grid is a non-changing feature of all LUCAS surveys, the location of each point remains the same throughout the years. Thus any discrepancy between the recorded theoretical location of a LUCAS point in the micro data and the grid must be corrected. This was done for all but 64 points from 2006 which where recorded on an inaccurate location and were thus removed from the grid.

To further asses spatial accuracy of the data, the distance between the theoretical point from the LUCAS grid (th_long, th_lat), and the actual GPS measurement of the survey observation point (gps_lon, gps_lat) were compared. This is important for several reasons - firstly, it allows to ascertain the real distance between the point actually surveyed and the point supposed to be surveyed, which is, in a sense, a proxy for the quality of the surveyed observation itself; secondly, it is an accuracy check of the surveyed distance between the theoretical point and the survey observation point, as collected by the surveyor, “as provided by the GPS (in m)” (column obs_dist), and the distance between the same points as calculated from the data (column th_gps_dist). It must be noted that for the 2006 survey the variable obs_dist was collected as a range, whereas for the other years it represents the actual value of the distance. Because of this lack of uniformity, it was decided to hard code the values for 2006 to match exactly with the calculated distance. In this way we ensure consistency in the data type of the column, yet sacrifice the nuances from changing the original data. The procedure explains that, in 2006, we see a 100% match between recorded and calculated distance (Table 7), whereby for 2009 a match of 96.3%, meaning that for only 3.7% of the cases did the value not match. In carrying out this comparison it became apparent that the percentage of matching distances has increased throughout years probably due to better precision of positioning sensors. Thus the total amount of error in 2018 is reduced to a negligible 0.31%. Furthermore, the comparison was instrumental in the flagging and removing of a number of records that have inaccurate GPS coordinates most probably due to sensor malfunction. Cross-checking with the source data, we found that the error is indeed present in the source data, rather than introduced during processing - something which would have been hard to spot otherwise. The distribution of these calculated distances, alongside an equivalent distribution of the surveyed distances, can be found in Fig. 6. The distance between 75% of the points (1–3 quantile) is between 1.1 and 21.2 meters, meaning that only a fourth of the points have a distance greater than this. For the surveyed distances the ranges are similar - 75% of the values fall between 1.0 and 30.0 meters. From the distributions we see that there is a lot more nuance in the values of the calculated distances, which makes sense as they are represented by numbers with decimals, which have a lower frequency than the integers, representing the surveyed distances. The values shown in the red part of the histogram of surveyed distances represent the values from 2006, which are copied from the calculated distance in order to hard code a numerical in the place of the categorical value of the variable in the source data. The theoretical grid of LUCAS point location is stable over time. However, according to the survey conditions and the terrain and accuracy of the GPS positioning, the surveyor may not be able to reach the point. This results in effective variations of the position of the observer through time (Fig. 7).

Table 7 Percentage (%) of points for which the distances between the theoretical point from the LUCAS grid (th_long, th_lat) and the actual GPS measurement (gps_lon,gps_lat) taken during surveying and calculated post factum match or not.
Full size table
Fig. 6
figure 6

Comparison of distributions between (a) calculated distances and (b) surveyed distances between LUCAS theoretical points and actual GPS position of surveyor. The red-colored part of the distribution in subfigure (b) represents the data from 2006, which is copied from the calculated distances (th_gps_dist).

Full size image
Fig. 7
figure 7

Stability of points and location change over time as illustrated: (a) Example of a surveyed point (id 40402278) at close distance (<2 m) and (b) Example of a surveyed point (id 63861648) at large distance (1938m). Location change can be either because of survey conditions, the accessibility of the terrain, and/or accuracy of GPS positioning. The background RGB imagery is obtained from “Map data ©2019 Google”.

Full size image

In addition to the theoretical grid and survey point location, this data descriptor provides the East-facing transect geo-location data. No additional geo-located spatial information is collected in the transect module and this is probably a shortcoming in the survey design resulting from trade-offs between the cost of the survey and its objectives. The theoretical transect line (with the same geometry as the one provided with this data descriptor) is displayed on the ground document of the surveyor. The surveyor has then to walk on the line and to record the successive land cover and landscape elements as described in the survey methodology. The only geo-location accuracy information relevant for the transect module is thus the same as presented previously, i.e. distance between the theoretical point and the GPS measured surveyed point. Then the successive land covers and landscapes surveyed along the 250-m line are collected as a sequence without distance or geo-located information.

Usage Notes

To summarize, the work documented in this data descriptor consists of49: (1) Multi-year harmonised LUCAS table, (2) Archive with high resolution LUCAS photos, (3) LUCAS survey geometries and point locations, (4) R package51, (5) Data descriptor of resulting database and (6) a Documentation table for users to quickly identify the differences of collected data between LUCAS campaigns micro-data and harmonised database.

The harmonised LUCAS product reduces the complexity and layered nature of the original LUCAS datasets. In doing so, it valorizes the effort of many surveyors, data cleaners, statisticians, and database maintainers. The database’s novelty lies in the fact that for the first time, users can query the whole LUCAS archive concurrently, allowing for comparisons and combinations between all variables collected during the relevant reference years. The homogeneity of the product facilitates the unearthing of temporal and spatial relations that were otherwise jeopardized by the physical separation between survey results. Moreover, by avoiding the burden of combing through the cumbersome documentation, the user is now free to concentrate on the research, thereby facilitating scientific discovery and analysis. Naturally, the product suffers from the shortcomings inherent in the source data, such as any inadequate surveying, surveyor or technology-related errors of precision while taking coordinates or measurements, etc. The harmonisation process itself also reveals some inconsistencies in the source data. For instance, certain variables could not be harmonised between survey years. These are mostly related to measurements of percentage or extent of coverage. Where in the early stages of LUCAS surveyors were asked to fill in a multiple choice questionnaire, listing a range of values, in subsequent surveys the surveyor was asked to fill in the actual value in quantified units. This situation applies mostly, though not exclusively, to the 2006 survey, which makes it impossible for these variables to be translated into the user friendly version; therefore in these cases the variables of 2006 must remain in their original coding. Additional information can be found in the comments section of the record descriptor.

Another shortcoming is the change of hierarchy of the LUCAS classification system between the different surveys, mainly concerning LC/LU, as well as LC and LU types. A table is provided to document this shortcoming (see special remarks in the Table (“LC (LU) changes” in the file LUCAS-Variable_and_Classification _Changes.xlsx49).