KR20150085716A - Setup Method for Web Scraping Data Extraction - Google Patents

Abstract

The present invention relates to a setup method for web scrapping extracted data, which comprises the following: a step of program operation (S100) that operates the GUI based web scrapping setup program (100) that outputs an action element list window (110), a scenario board window (120) showing action elements selected by a user, and an action element attribute input window (130) that outputs and selects the attribute for a web browser, and the web browser (140); a step of action element selection (S200) where a user presses the action element button in the action element list window (110) to register in the scenario board window (120); step of scenario preparation (S300) where a user selects and inputs the data and attribute displayed on the current page of the web browser (140) by using the action element registered in the scenario board window (120) in the step of element selection (S100) and then prepares a scenario for data extraction; and a step of setup saving (S400) where a user saves the data extracted in the step of data extraction (S300) and the scenario setup data as a file. Using the scenario setup file saved in the step of setup saving (S400), the web scrapping main program′s data extraction setup is added, deleted or changed. According to the present invention, the web scrapping or web automation related main program′s data extraction setup can be easily changed or configured on the GUI by using the attribute addition method.

Description

How to Set Web Scraping Extraction Data {Setup Method for Web Scraping Data Extraction}

More particularly, the present invention relates to a Web scraping extraction data setting method that automatically collects and extracts necessary information from a web site, compares the collected information, and provides a web scraping extraction data setting method .

Web scraping refers to a technique or program that automatically accesses the Web system by a program and displays the data on the screen, and then extracts and retrieves the necessary data.

The data extracted by web scraping can be stored in other programs or databases, and can be browsed or used at any time when needed, and utilized as comparative analysis data using the extracted data.

Such web scraping programs are the core of technologies for collecting necessary information on a web site, a technique for converting to a certain format, and a scraping driving technology. These web scraping programs are installed in a server, installed in a client, Or a combination of the server and the client.

This web scraping technique is mainly used for collectively inquiring information on individual financial assets or for integrating and managing transaction details or collecting information such as travel agencies, airlines, hotels and logistics services, weather, and news It is also used as an application.

Conventional web scraping programs are implemented by automation systems that use a http protocol layer or a built-in web browser by the programmer to put workforce for repetition or data collection of a web site.

However, in order to collect (extract) the data, even if parsing the source html code using string parsing or regular expression, the problem of not collecting data when the design of the target website is changed In this case, the code of the web scraping program must be rewritten. These operations require a professional programmer, and therefore, it is difficult for the user to directly modify the web scraping code when necessary.

Also, this method has a problem that the readability of the program code is poor and the debugging is very inconvenient.

KR 10-2013-0094935 A KR 10-2010-0132576 A KR 10-2002-0068566 A KR 10-0633534 B1

The present invention has been made in order to solve the problems of the conventional Web scraping extraction data setting method as described above. The present invention does not require a professional programmer when creating web scraping or web automation related programs, The purpose of this paper is to provide a web scraping extraction data setting method that can create a professional web scraping program by simply creating a scenario through a method.

It is an object of the present invention to provide a Web scraping data extracting method that includes an action element list window including a navigation button, a URL extracting button, a data extracting button, and an attribute value extracting button, a scenario in which a currently selected action element is output by a user, A program driving step of driving a GUI type web scraping setting program in which an action element attribute input window and a web browser are displayed in which an attribute of a web browser is selected and outputted according to an action element registered in a board window and a scenario board window; An action element selection step of the user selecting an action element button of the action element list window and registering the action element button in the scenario board window; A scenario creation step of creating a data extraction scenario by selecting and inputting data and attribute values output on a current page of the web browser by the action elements registered in the scenario board window in the action element selection step; And a setting storing step of storing the data extracted in the scenario creating step and the scenario setting information in a file, and the data extraction setting of the web scraping main program is added, deleted or changed using the scenario setting file stored in the setting storing step ≪ / RTI >

At this time, in the setting storing step, the scenario setting file may be stored as an executable file that can be executed alone, and the user may execute the scenario setting file alone to change the scenario setting of the main program.

The scenario creating step includes a single data extracting step of extracting one piece of data output to the current page of the web browser, a plurality of data extracting steps of extracting a plurality of pieces of data described in the table among the current pages of the web browser, Element ID, tag name and offset, and attribute value extraction step in which X, Y coordinate data is extracted.

In addition, when the text data of the reference point selected by the user is extracted using the DOM index, the single data extracting step and the plural data extracting step can be performed by automatically extracting the data of the subtree of the selected reference point text data.

The navigation button is an action element for selecting a web browser from which scenario data is to be extracted. The URL extraction button is an action element that automatically extracts a URL address to be guided from the current page of the selected web browser to the next step. An action element for extracting data of a current page of the web browser, and an attribute value extraction button is an action element for extracting each attribute value selected from an element ID, a tag name and an offset, and X and Y coordinates of a current page of the web browser by a user Element. ≪ / RTI >

The present invention does not require the help of a professional programmer to change the source code of the main program when changing data extraction settings (scenarios) of web scraping or web automation related programs, By using the attribute addition method, a desired data extraction method can be simply set.

In addition, since the scenario setting file (binary file, script file) set is stored in a format that can be executed in a single executable format, the present invention allows the user to execute the scenario setting file on a computer (PC) having a main program (Web scraping program) The setting is added or changed.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an example of a web scraping extraction data setting method according to the present invention;
2 is a block diagram illustrating an example of a driving screen of a web scraping setting program according to the present invention.
FIG. 3 is a diagram showing an example of an action element attribute input window in which a navigation action element button according to the present invention is registered in a scenario board window;
FIG. 4A is a diagram showing an example of an action element attribute input window in which a URL action element button according to the present invention is registered in a scenario board window; FIG.
4 (b) is a diagram showing an example in which URL data is extracted and stored in FIG. 4 (a)
5 is a diagram showing an example of a scenario creation step according to the present invention;
FIG. 6A is a diagram illustrating an example of an action element attribute input window in which a single data extraction action element button according to the present invention is registered in a scenario board window; FIG.
6 (b) is a diagram showing an example in which single data is extracted and stored in FIG. 6 (a)
FIG. 7 is a diagram showing an example of an action element attribute input window in which a plurality of data extraction action element buttons according to the present invention are registered in a scenario board window;
FIG. 8 is a diagram showing an example of an action element attribute input window in which an attribute value extraction action element button according to the present invention is registered in a scenario board window;
9A and 9B are block diagrams illustrating an example of an action element attribute input window in which an attribute value extraction option according to FIG. 8 is selected.

Hereinafter, the structure and operation of the present invention will be described in more detail with reference to the accompanying drawings, which show preferred embodiments.

The present invention relates to an action corresponding to an action performed on a web browser by using a GUI so that repetitive actions, surfing, data extraction, etc. of a web page requiring web scraping or a web automation program can be performed without assistance of a professional programmer (S100), an action element selection step (S200), a scenario creation step (S300) as shown in FIG. 1, and a method of constructing a scenario by adding an action element And a setting storing step (S400). Each step will be described in detail below.

(1) program driving step (S100)

This step is a step of driving the web scraping setting program 100 on the user's PC when addition, deletion or change of the data extraction setting of the web scraping or web automation related program (hereinafter referred to as 'main program') is requested, The web scraping setting program 100 is programmed and executed singly in any one of computer programming languages such as commonly known C language, ASP, PHP, JSP, and the like.

When the user executes such a web scraping setting program 100, a computer monitor (not shown) is provided with an action element list window 110, a scenario board window 120, an action element property window The web browser 130 and the web browser 140 are respectively displayed on the main screen in a graphical user interface (GUI) manner.

At this time, the action element list window 110 includes a navigation button 111, a URL extracting button 112, a data extracting button 113, and an attribute value extracting button 114. The navigation button 111 displays the scenario data Extracting button 112 is an action element for automatically extracting a URL address to be guided from the current page of the selected web browser to the next step and the data extracting button 113 is an action element for selecting a web browser to be extracted, The attribute value extraction button 114 extracts each attribute value selected from the element ID, tag name, offset, and X, Y coordinates of the current page of the web browser by the user The action element, and a detailed description of the above action elements will be described later.

When the user clicks a mouse button or drag-and-drop an action element button with the mouse, the action element is simply registered in the scenario board window 120, The operation of the corresponding content is performed in the action element, and the data extraction scenario is freely added, deleted or changed.

The action element property window 130 has a function of activating the action element selected by the user of the scenario board window 120 and guiding the currently active action element to the user. Data, multiple data, and web browser property values of the current page, and at the same time guiding the user to select or input an attribute as needed.

(2) Action element selection step S200

In this step, the user drives the web scraping setting program 100 by the program driving step S100, and then selects one of the action element buttons of the action element list window 110 displayed on the main screen, And registers it in the window 120.

At this time, the corresponding attribute value of the current page of the web browser 140 is automatically output to the action element property window 130 according to the action element selected by the user. For example, when the navigation button 111 is displayed on the scenario board window 120 Once registered, a list in which the current page address (URL) of the web browser 140 is output is output to the action element attribute window 130 as shown in FIG.

When the URL extracting button 112 is registered in the scenario board window 120, the list for automatically extracting the link URL of the current page of the web browser 140 as shown in FIG. 4A is an action element attribute The URL is automatically extracted and stored as shown in FIG. 4 (b) when the user selects a link button (link 1, link 2) output to the web browser 140 at this time .

When the navigation button 111 and the URL extracting button 112 are selected, the user does not create a scenario but extracts the information of the current page of the web browser 140, so that a scenario creating step S300 Instead, the action element selection step S200 for returning to the action element is returned by the loop syntax.

On the other hand, when the user selects the data extraction button 113 and the attribute extraction button 114 and registers them in the scenario board window 120, an option list such as a dialog box, a check box, a button, The process proceeds to step S300. In this case, when the user selects the reference point data (string, matrix, coordinates, etc.) corresponding to the output option list, the scenario is created by a method in which the data of the corresponding condition is automatically input .

When the scenario creation is completed in the scenario creation step S300, the process returns to the action element selection step S200 and the main screen is returned by the loop syntax so that the action element can be reselected, Elements are selected in multiple redundancy to create various scenarios.

(3) Scenario creation step (S300)

In this step, in the action element selection step S200, the user clicks or drags the action element button and registers it in the scenario board window 120, extracts necessary data and attributes from the current page of the web browser 140 (Setting) to be driven by the Web scraping program.

The scenario creation step S300 includes a single data extraction step S310 for extracting one piece of data output to the current page of the web browser as shown in FIG. 5, a plurality of A plurality of data extracting steps S320 for extracting data, and an attribute value extracting step S330 for extracting a current page element ID, a tag name, an offset, and X, Y coordinate data of the web browser.

In this case, when the text data of the reference point selected by the user is extracted using the DOM index, the data of the subtree of the selected reference point text data is automatically extracted in the single data extracting step S310 and the plural data extracting step S320.

In the conventional web scraping program, a regular expression or an extraction method using string indexing is used for data extraction. In this case, data modification is difficult and data extraction performance is not ensured.

Accordingly, in the present invention, data is extracted by the DOM index method. When the user selects the reference point text of the DOM tree and the offset of the tag, data is extracted based on the offset of the reference point text and the tag , Thereby ensuring stable data extraction capability while facilitating data modification.

Here, the DOM (Document Object Model) index refers to a method of representing a structured document as an object-oriented model. Such a DOM is classified and stored in a tree format, so that the parent tree data of the DOM tree or the tree When one piece of data is selected as the reference point text by the user, it is possible to automatically extract other pieces of data in the corresponding tree. In the case of extracting a single piece of data, extracting a plurality of pieces of data, , And the case of the case in which the case is described.

(a) When extracting a single data

When the user selects a single data extraction button 113A among the data extraction buttons 113 and extracts single data output to the web browser 140, an action element property window (Fig. 6 130, a list of reference point texts, tags, offsets, names, and the like are output, and an input window (without a reference numeral) for inputting data selected by the user is output to each list. In each of the reference point text input window and the offset input window, a button (without a designation code) that the user can click and select with a mouse is outputted. In this case, the user first clicks the button of the reference point text input window with the mouse, Then, when a character string (telephone number, name, address, etc.) reference point text output from the current page of the web browser 140 is selected, the reference point text is automatically input to the corresponding input window.

After inputting the reference point text, the user again clicks the offset input window button to change the input state of the mouse, and when the input value of the selected string is selected again from the reference point text, the tag of the action element property window 130 And the corresponding information is automatically input to the offset input window.

After the information of the reference point text, the tag, and the offset are respectively input, the user selects the name input window and inputs the name of the corresponding data, and the data is stored in the programming syntax as shown in FIG. 6B. Then, the corresponding information is temporarily stored, and then returns to the main screen to select another action element.

(b) When extracting multiple data

7, when the user selects a plurality of data extraction buttons 113B among the data extraction buttons 113 and extracts a plurality of data output to the web browser 140, the action element property window 130 is displayed, A list of the reference point text, the tag, the offset, the name, and the like is output. In a single data output method, the user first activates the mouse by clicking the reference point text input window button, One piece of data is selected as the reference point data, thereby automatically inputting the reference point data selected by the user in the reference point text input window.

Then, when the data to be extracted is selected from the table or the form data by clicking the offset input window button, data of the longitudinal and lateral columns are automatically extracted based on the column, and the tag and offset input window are automatically input. And enter the unified name of the extracted data.

In addition, the user may select and store only the data of a specific column among the entire data automatically extracted in the extended property input window. In this case, the user may click the button (no reference numeral) provided in the extended property input window, By selecting some data in the column, the data of the column is automatically extracted and input.

(c) When extracting attribute values

When the user selects the attribute value extracting button 114 and extracts the attribute value output to the web browser 140, an element ID, a tag name and an offset are displayed in the action element property window 130 as shown in Fig. An X, Y coordinate input window (without a reference numeral) is output, and a button (without a reference numeral) that can be selected by the user is output to each input window.

At this time, as shown in FIG. 9A, the element ID, the tag name, and the offset input window include a character string for automatically inputting input information such as an ID and a password PW requested by the web browser 140 And coordinate values such as a login button and the like are input to the X, Y coordinate input window.

When the web scraping main program is driven by these attribute values, the data collection activity is performed while automatically inputting the user's login information and the like. Thus, the login information pre- And the method of automatically accessing the web site is already well known, so a detailed description thereof will be omitted.

As described above, according to the present invention, the reference point data of the parent tree is selected from the data stored in the DOM index tree system, and some data in the tree is selected, thereby extracting part or all of the data in the tree.

(4) Setting storage step (S400)

This step is a step of storing (exporting, exporting) extracted data and scenario setting information before the user exits from the main screen and finishes the program, when the scenario creation is completed through the scenario creation step (S300) If scenario creation is completed and the user selects to save the scenario setting information, the scenario setting information is stored as a binary file and a script file. In the present invention, the scenario setting information is formatted and stored as an EXE file of a window that can be executed independently.

Accordingly, in the present invention, the finally saved scenario setting information file is independently executed in the PC on which the web scraping main program is installed, so that the data extraction setting of the web scraping main program is added or changed.

When a scenario (setting) for extracting data is added or changed in a web scraping or web automation related program, the present invention can be applied to a web scraping setting program The data extraction setting can be easily changed or added by selecting the action element using the GUI and adding the attribute.

Since the scenario setting information is stored as a single executable file, the user can execute the scenario setting file on the computer (PC) and copy the separate single binary file and the script file according to the scenario set in the main GUI program, If you modify the exported scenario, if the script file is loaded from the main GUI program, the scenario saved in the script file will be loaded and you will be able to modify the exported scenario again. Export and distribute.

100: Web scraping setup program 110: Action element list window
111: navigation button 112: URL extraction button
113: Data extraction button 113A: Single data extraction button
113B: Multiple data extraction button 114: Attribute value extraction button
120: Scenario board window 130: Action element property input window
140: Web browser

Claims (5)

An action element list window 110 including a navigation button 111, a URL extracting button 112, a data extracting button 113 and an attribute value extracting button 114; An action element attribute input window 130 in which an attribute of a web browser is selected and outputted according to an action element registered in the scenario board window 120 and a GUI A program driving step (SlOO) for driving the web scraping setting program 100 of the method;
An action element selection step (S200) for the user to select an action element button of the action element list window 110 and register the selected action element button in the scenario board window 120;
In the action element selection step S100, data and attribute values output on the current page of the web browser 140 are selected and input by the action elements registered in the scenario board window 120 to create a data extraction scenario A scenario creation step (S300);
And a setting storing step (S400) for storing the data extracted in the scenario creating step (S300) and the scenario setting information as a file,
Wherein the data extraction setting of the web scraping main program is added, deleted, or changed using the scenario configuration file stored in the setting storage step (S400).
The method according to claim 1,
In the setting storage step (S400), the scenario setting file is stored as an executable file that can be executed independently,
Wherein the user executes the scenario setting file alone to change the scenario setting of the main program.
The method according to claim 1 or 2,
The scenario creation step S300 includes a single data extraction step S310 for extracting one piece of data output to the current page of the web browser; A plurality of data extracting steps (S320) of extracting a plurality of data described in a table among the current pages of the web browser, an attribute value extracting step (S320) of extracting a current page element ID, a tag name, an offset, and X, Y coordinate data of the web browser (S330). ≪ / RTI >
The method of claim 3,
In the single data extracting step S310 and the plural data extracting step S320, when the text data of the reference point selected by the user is extracted using the DOM index, the data of the subtree of the selected reference point text data is automatically extracted A method for setting a web scraping extraction data.
The method according to claim 1,
The navigation button 111 is an action element for selecting a web browser from which scenario data is to be extracted. The URL extraction button 112 is an action element for automatically extracting a URL address to be guided from the current page of the selected web browser to the next step, And the attribute extraction button 114 extracts an element ID, a tag name and an offset of the current page of the web browser by the user, and X (x, y) are the action elements for extracting data of the current page of the web browser. , And Y coordinates of each of the plurality of action elements.

Images