KR20240082672A - Scraping support server, method for scraping service and computer program for the same - Google Patents

Abstract

The scraping support server includes a communication unit configured to receive an information request for scraping one or more pieces of information from a user device; and a data processing unit configured to, in response to the information request, generate input data for the user device to scrape the one or more information from the information provider server, based on communication policy information of the information provider server. You can. At this time, the communication unit is further configured to transmit the input data to the user device. When using the scraping support server, when the user device performs scraping, data modification is processed on the server side, which is relatively easy to modify and distribute, so the scraping engine is modified to respond to each change and distributed to the user device. There is no need to do so, and there is an advantage that scraping technology can be used regardless of the type of user device platform (e.g., Android, iOS, etc.).

Description

Scraping support server, scraping service method, and computer program therefor {SCRAPING SUPPORT SERVER, METHOD FOR SCRAPING SERVICE AND COMPUTER PROGRAM FOR THE SAME}

Embodiments relate to a scraping support server, a scraping service method, and a user device and computer program therefor. More specifically, embodiments relate to technology that allows scraping technology to be used without modifying the application even if changes occur by transmitting data for the operation of a scraping engine running on a user device such as a mobile terminal from the server side to the user device. It is about.

Scraping refers to a technology in which a computer system, such as a user terminal or server, connects to a specific server on behalf of the user, performs user authentication, and collects necessary information. Scraping technologies include screen scraping, which collects information on the screen after the computer system accesses a web page, etc. to display the necessary screen, and an application programming interface (API) that is defined in advance by the server. There is a data scraping technology that receives information from a server through .

In addition, scraping technology can be either a mobile scraping method in which scraping is executed through an application (or app) on a mobile device such as a user's smartphone, or a scraping engine running on a server, depending on the type of computer system that performs it. It can be classified into server-type scraping. Recently, the mobile scraping method is increasingly being chosen instead of the server-type scraping method, which incurs high costs in service operation.

However, when using the mobile scraping method, since the scraping engine is mounted on the mobile application, there is a disadvantage that the application must be redistributed if the input/output data format for scraping changes or the data format needs to be modified.

1 is a flowchart showing a method of modifying a scraping application according to the prior art. Referring to FIG. 1, conventionally, a distribution server 1 that provides an application store service, etc. distributes an application equipped with a scraping engine to a user device 2, such as a mobile terminal (S11), and the user device 2 distributes it. Information was scraped from the information provider server (3) using the method (S12).

However, if the communication policy of the information provider server 3, such as the connection requirements for the information provider server 3 or the type of information provided by the information provider server 3, is modified (S13), the policy before modification is changed. Information scraping through the scraping engine developed as a standard fails (S14). To solve this problem, the service provider must develop a new scraping engine that matches the changed policy of the information provider and distribute it to the user device (2) (S15), and the user device (2) downloads the newly distributed scraping engine. Scraping of information becomes possible only when used (S16).

However, there is a problem in that re-editing and distributing the application every time a change in the information provider's policy occurs, causing a burden in terms of cost and manpower to the business providing the service. Additionally, when an application is distributed through an application store, the application inspection process by the store provider takes time, so there is a problem that the user's use of the service is delayed during the inspection.

Registered Patent Publication No. 10-1402146

According to one aspect of the present invention, the server side, which is easy to modify and distribute data, transmits data for the operation of a scraping engine running on a user device such as a mobile terminal, thereby changing the communication policy of the information provider, etc. Even if this occurs, a scraping support server that allows scraping technology to be used without modification of the application, a scraping service method, and a user device and computer program for the same can be provided.

A scraping support server according to one aspect of the present invention includes a communication unit configured to receive an information request for scraping one or more information from a user device; and a data processing unit configured to, in response to the information request, generate input data for the user device to scrape the one or more pieces of information from the information provider server, based on communication policy information of the information provider server. . At this time, the communication unit is further configured to transmit the input data to the user device.

In one embodiment, the communication unit is further configured to receive output data of the information provider server corresponding to the input data from the user device. At this time, the data processing unit is further configured to extract the one or more pieces of information from the output data based on communication policy information of the information provider server.

In one embodiment, the communication unit is further configured to transmit the input data to the user device and receive the output data from the user device through a preset application programming interface (API).

The scraping support server according to one embodiment further includes a storage unit configured to store the one or more information extracted by the data processing unit. At this time, the communication unit is further configured to provide the one or more information stored in the storage unit to the user device.

In one embodiment, the input data includes an information authentication unit for authentication to the information provider server, a scraping command code defining a scraping rule, and one or more scraping request data defining a data item to be scraped.

In one embodiment, the output data includes an authentication result portion indicating an authentication result for the information provider server, a scraping command code defining a scraping rule, and one or more scraping result data defining a scraping result of a data item. .

A user device for a scraping service according to one aspect of the present invention includes a memory storing executable instructions; A communication module configured to communicate with a user device; and a processor communicatively connected to the memory and the communication module. At this time, by executing the instruction, the processor transmits an information request for scraping one or more pieces of information to a scraping support server, receives input data corresponding to the information request from the scraping support server, and converts the input data into information. It is configured to receive output data from the information provision agency server in a scraping manner by transmitting it to the provision agency server.

In one embodiment, the processor is further configured to transmit the output data to the scraping support server and receive the one or more information extracted from the output data from the scraping support server.

The scraping service method according to one aspect of the present invention includes the steps of a scraping support server receiving an information request for scraping one or more information from a user device; generating, by the scraping support server, input data for the user device to scrape the one or more pieces of information from the information providing agency server, based on communication policy information of the information providing agency server; and transmitting, by the scraping support server, the input data to the user device.

In one embodiment, the step of transmitting the input data to the user device includes the step of the scraping support server transmitting the input data to the user device through a preset API.

The scraping service method according to one embodiment includes the steps of receiving, by the scraping support server, output data of the information provider server corresponding to the input data from the user device; and extracting, by the scraping support server, the one or more pieces of information from the output data based on communication policy information of the information provider server.

In one embodiment, the step of receiving the output data includes the step of the scraping support server receiving the output data from the user device through a preset API.

The scraping service method according to an embodiment includes the steps of: the scraping support server storing the one or more pieces of information extracted from the output data; And the scraping support server further includes providing the one or more information stored in the scraping support server to the user device.

A scraping service method according to another aspect of the present invention includes the steps of a user device transmitting an information request to scrape one or more information to a scraping support server; receiving, by the user device, input data corresponding to the information request from the scraping support server; transmitting, by the user device, the input data to an information provider server; and receiving, by the user device, output data from the information provider server in response to the input data through a scraping method.

The scraping service method according to one embodiment includes the user device transmitting the output data to the scraping support server; and receiving, by the user device, the one or more information extracted from the output data from the scraping support server.

The computer program according to one embodiment is combined with hardware to execute the scraping service method according to the above-described embodiments, and may be stored in a computer-readable medium.

According to the scraping support server, user device, and scraping service method according to one aspect of the present invention, the scraping support server transmits input and output data for the operation of a scraping engine running on a user device such as a mobile terminal to the user device, Even if there is a change in the communication policy of the information provider, processing of input/output data packets in accordance with the changed policy is performed through the scraping support server.

Through this, data modification can be processed on the server side, where it is relatively easy to modify and distribute data, and the scraping engine is modified and distributed to user devices to respond to all changes involved in realizing the scraping function. The advantage is that you don't have to.

In addition, since the processing of input and output data for the operation of the scraping engine is performed on the scraping support server, the advantage is that the scraping technology can be used on any computing device regardless of the type of user device platform (e.g. Android, iOS, etc.). there is.

Furthermore, even if the data format that the application inputs and outputs is modified, it does not require an inspection process by the application store operator, and the changes are immediately reflected in the operation of the application through communication with the scraping support server, ensuring immediacy of service. There is an advantage.

1 is a flowchart showing a method of modifying a scraping application according to the prior art.
Figure 2 is a schematic block diagram showing the configuration of a scraping service system according to an embodiment.
Figure 3 is a schematic block diagram showing the hardware configuration of a user device according to an embodiment.
Figure 4 is a flowchart showing each step of the scraping service method according to an embodiment.
Figures 5 and 6 are conceptual diagrams showing the structure of input/output data used in a scraping service method according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Figure 2 is a schematic block diagram showing the configuration of a scraping service system according to an embodiment.

Referring to FIG. 2, the scraping service system according to this embodiment includes a scraping support server 5 configured to operate in communication with the user device 2. Also in one embodiment, the scraping service system may further include a user device 2.

The scraping support server 5 may receive an information request for scraping from the user device 2 and provide input data necessary for the user device 2 to scrape information from the information provider server 3. In addition, the scraping support server 5 may extract information corresponding to the output data received from the information providing agency server 3 and provide it to the user device 2 or store the information.

The user device 2 is a device used by a user who wishes to receive information from the information provision agency server 3. The user device 2 is configured to receive information by scraping from the information provider server 3 using data received from the scraping support server 3. In addition, the information provider server 3 is any organization that provides user-related information, and may be a financial institution such as a bank, credit card company, securities company, or insurance company, or a public institution such as the National Tax Service, government offices, the government, or a local government. It is not limited.

The scraping support server 5 is configured to operate and communicate with the user device 2 via a wired and/or wireless network 4. Additionally, the user device 2 is configured to scrape information from the information provider server 3 via a wired and/or wireless network 4 .

For example, the wired and/or wireless network (4) between the scraping support server (5), the user device (2), and the information provider server (3) may include a Local Area Network (LAN), a Metropolitan Area Network (MAN), GSM (Global System for Mobile Network), EDGE (Enhanced Data GSM Environment), HSDPA (High Speed Downlink Packet Access), W-CDMA (Wideband Code Division Multiple Access), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access) Access), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE) , Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability It may refer to a communication network using one or more communication methods selected from the group consisting of (for Microwave Access) and ultrasonic communication, but is not limited thereto.

The number and form of user devices 2 shown in FIG. 1 are merely exemplary. For example, in FIG. 1, the user device 2 is shown in the form of a smartphone, but in other embodiments, the user device 2 may be other types of mobile communication terminals, personal computers (PCs), It may be implemented in the form of any computing device, such as a laptop computer, a personal digital assistant (PDA), a tablet, or a set-top box for an IPTV (Internet Protocol Television).

The scraping support server 5 may communicate with an application running on the user device 2 and configured to scrape information, thereby providing data to enable operation of the application.

In one embodiment, the scraping support server 5 includes a communication unit 51 and a data processing unit 52. Also, in one embodiment, the scraping support server 5 may further include a database (DB) 50. Furthermore, in one embodiment, the scraping support server 5 may further include a storage unit 53. Meanwhile, each of these units 51 to 53 or the DB 50 may be realized at least partially using the hardware of the scraping support server 5.

That is, the scraping support server 5 according to embodiments and each component included therein may be entirely hardware, or may be partially hardware and partially software. For example, each component of the scraping support server 5 may collectively refer to hardware and software related thereto for processing or/or exchanging data of a specific format and content through electronic communication. In this specification, terms such as “unit,” “module,” “device,” “terminal,” “server,” or “system” are intended to refer to a combination of hardware and software driven by the hardware. For example, hardware may be a data processing device that includes a CPU or other processor. Additionally, software driven by hardware may refer to a running process, object, executable, thread of execution, program, etc.

Additionally, each element constituting the scraping support server 5 is not necessarily intended to refer to a separate device that is physically separate from each other. That is, each unit 51 to 53 and DB 54 of the scraping support server 5 shown in FIG. 1 functionally divide the hardware constituting the scraping support server 5 according to the operation performed by the hardware. However, each part does not necessarily have to be provided independently from each other. Of course, depending on the embodiment, one or more of the above-described units 51 to 53 and the DB 54 may be implemented as separate devices that are physically separate from each other.

The communication unit 51 may receive an information request for scraping from the user device 2, generate input data for scraping in response to the information request, and provide it to the user device 51. Additionally, the communication unit 51 may further receive output data scraped by the user device 2 from the information provision agency server 3. Furthermore, the communication unit 51 may transmit one or more information extracted from the output data to the user device 2.

In one embodiment, an application for scraping running on the user device 2 and the scraping support server 5 may communicate with each other through a preset application programming interface (API). That is, the communication unit 51 of the scraping support server 5 can receive an information request from the user device 2 through the API, and also sends input/output data for scraping and/or information extracted from the output data through the API. It can be transmitted to the user device (2).

The data processing unit 52 may generate input data to be transmitted to the information provider server 3 when the application on the user device 2 performs a scraping operation. Additionally, the data processing unit 52 may extract one or more information that the user wishes to inquire from the output data scraped by the user device 2 from the information provision agency server 3.

Specifically, the data processing unit 52 extracts one or more data items included in the information request received from the user device 2 and sends the data to the information provider based on the communication policy information of the information provider holding the data item. You can create a data packet to request. The data packets generated in this way are referred to herein as input data. The input data is transmitted by the user device 2 to the information provider server 3 in order for the application on the user device 2 to scrape information from the information provider server 3 .

The application on the user device 2 receives the scraping result from the information provider server 3 by transmitting the input data received from the scraping support server 5 to the information provider server 3, and as described herein, the user Data received by device 2 is referred to as output data. In one embodiment, the data processing unit 52 may extract data items that the user wants to scrape from the output data received by the user device 2, based on communication policy information of the relevant information provider.

For the above operation, communication policy information of one or more information providing organizations may be stored in the DB 50 of the scraping support server 5. Additionally, the format or composition of input/output data for scraping may vary depending on the communication policy of the information provider. In this specification, communication policy information refers to the type of communication protocol, the format and/or size of data packets transmitted and received, the writing method of information in the data packet in the communication process for receiving information from the information provider server 3, It refers to arbitrary matters affecting communication with the information provider server (3), such as type and order, and is not necessarily limited to those related to the configuration of the communication session.

In one embodiment, the data packet corresponding to the input data includes an information authentication unit for authentication to the information provider server 3, a scraping command code defining a scraping rule, and one or more scraping requests defining the data items to be scraped. Can contain data. In addition, the data packet corresponding to the output data includes an authentication result section indicating the user authentication result at the information provider server 3 corresponding to the packet configuration of the input data, the scraping command code, and a scraping result of the data item. It may contain one or more scraping result data.

Examples of data packets processed by the scraping support server 5 according to embodiments will be described in detail later with reference to FIGS. 5 and 6.

Conventionally, as described above with reference to FIG. 1, when the communication policy of an information providing organization changes, an application for scraping must be modified and distributed to the user device 2. However, using the embodiments of the present invention, the input data for scraping of the information provider server 3 and the output data from the information provider server 3 are processed through the scraping support server 5 and sent to the user device ( Since it is provided in 2), there is an advantage that there is no need to modify and distribute the application every time the communication policy of the information provider changes.

In one embodiment, the scraping support server 5 may store one or more information extracted from the output data in the storage unit 53. In this case, the user can check his or her scraped information by communicating with the communication unit 51 of the scraping support server 5 using the user device 2. However, in another embodiment, the scraped user information may be stored only on the user device 2 rather than on the server, and in this case, the storage unit 53 may be omitted.

FIG. 3 is a schematic block diagram showing the hardware configuration of a user device according to an embodiment, and shows the hardware configuration of the user device 2 described above with reference to FIG. 2 . However, it will be readily understood by those skilled in the art that the same type of hardware configuration can also be applied to a server device, for example, the scraping support server 5.

Referring to FIG. 3, the user device 2 according to embodiments is implemented in the form of a computing device including hardware 200, a memory 210, a processor 220, a communication module 230, and an input/output unit. It may include (240).

The memory 210 is a non-transitory computer-readable recording medium, and is a non-perishable large capacity device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It may include a permanent mass storage device. Here, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the above-mentioned device or server as a separate persistent storage device that is distinct from the memory 210.

Additionally, the memory 210 may store an operating system and at least one program code (for example, a code for a security module or an application installed to provide a specific service). These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards.

In another embodiment, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program is based on a computer program installed in memory ( 210).

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication module 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication module 230 may provide a function for the user device 2 to communicate with the scraping support server 5 and/or the information provider server 3 through a network. Additionally, the communication module 230 may provide functionality for the user device 2 to communicate with one or more other devices through a wired and/or wireless network. That is, the communication module 230 can realize each function of the user device 2 described throughout this specification by having its functions controlled by the processor 220 that refers to the memory 210.

The input/output unit 240 may be a means for interfacing with an external input/output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, and camera, and external output devices may include devices such as a display, speaker, and haptic feedback device. As another example, the input/output unit 240 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen.

Additionally, in other embodiments the user device 2 may include more hardware components than those shown in FIG. 3 depending on the nature of the device to which it is applied. For example, the user device 2 is implemented to include at least some of the above-described input/output devices or further includes other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. You may. As a more specific example, when the terminal device is a smartphone, the smartphone generally includes an acceleration sensor, a gyro sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration. It can be implemented to further include various components such as:

The scraping service method according to one embodiment may be performed by the user device 2 and/or the scraping support server 5 implemented in the form of a computing device including the hardware 200 configuration described above with reference to FIG. 3. there is. As an example, the scraping service method may be provided to the user in the form of a service based on at least one of applications, software, and other programs that operate based on the user device and/or server.

In one embodiment, the processor 220 transmits an information request for scraping one or more information to the scraping support server by executing instructions stored in the memory 210, and receives an input corresponding to the information request from the scraping support server. Can be operated to receive data.

Additionally, the processor 210 may receive output data from the information provision agency server through a scraping method by transmitting input data to the information provision agency server. To this end, an application corresponding to a so-called scraping engine is stored in the memory 210 of the user device 2 according to the embodiments, and the processor 210 executes this application, thereby eliminating the need for the user's manual work from the information provider server. You can collect the necessary information using a scraping method that does not involve scraping.

Furthermore, in one embodiment, processor 220 may transmit output data to a scraping support server. Additionally, the processor 220 may receive scraping result information extracted from the output data from the scraping support server.

Figure 4 is a flowchart showing each step of the scraping service method according to an embodiment.

Referring to FIG. 4, in one embodiment, the user device 2 may first receive authentication information for scraping from the user and store it (S21). At this time, input information is information that allows the computer system to verify the user's identity with the information provider on behalf of the user. For example, it may include a public certificate, joint certificate, ID and password, but is not limited to this. no.

In one embodiment, the authentication information is stored only on the user device 2 and is not transmitted to the scraping support server 5. By configuring in this way, the user's personal information can be prevented from being exposed to communication networks through the network and security can be maintained. Additionally, in this case, the authentication information may be stored in encrypted form. However, this is an example, and the method of handling authentication information in the scraping service method according to the embodiments is not limited to this.

When a user input requesting information required for scraping is input to the user device 2, the user device 2 may transmit a request for information for scraping to the scraping support server 5 (S22). The above transmission process may be performed based on a predefined API between the application running on the user device 2 and the scraping support server 2. That is, an application running on the user device 2 and equipped with a scraping engine can transmit data items requiring scraping to the scraping support server 5 through an API when it is necessary to execute a scraping operation.

The scraping support server 5 stores communication policy information related to one or a plurality of information provider servers 3, and when there is an information request from the user device 2, the information provider agency that needs to scrape the requested information Input data for scraping can be generated based on the communication policy information of the server 3 (S23). Additionally, the scraping support server 5 may transmit the generated input data to the user device 2 (S24).

In one embodiment, the scraping support server 5 may perform verification as to whether provision of information corresponding to the user's request can be realized through scraping before generating input data. For example, when a user requests scraping of specific information from a specific information provision agency server (3), but access to the corresponding information provision agency server (4) is impossible due to a failure, etc., or the information provision agency server ( In 4), there may be cases where the user does not provide the requested information, or the communication policy information of the relevant information provider server (4) cannot be confirmed.

In this case, the scraping support server 5 may provide the user device 2 with a response indicating that generation of input data is impossible instead of generating input data.

When the user device 2 receives input data from the scraping support server 5, the user device 2 transmits the input data to the information provider server 3 to obtain information from the information provider server 3. Scraping is possible (S25). The user device 2 may receive the information requested by the user through the above process through a scraping method from the information provider server 3, and store the received information in the user device 2 and/or provide it to the user. .

In one embodiment, the user device 2 may further transmit output data corresponding to the response of the information provider server 3 to the scraping support server 5 (S26). For example, there may be cases where interpretation of output data is not possible depending on the application mounted on the user device 2. In this case, the scraping support server 5 extracts one or more data items that the user wishes to scrape from the received output data (S27), and stores the extracted information in the scraping support server 5 or/or the user device ( 2) It can be transmitted (S28, S29).

However, in another embodiment, even when processing of the output data by the user device 2 is possible, the user device 2 may auxiliarily transmit the output data to the scraping support server 5. In this case, the information transmitted to the scraping support server 5 can be used to provide additional functions or information to users. For example, the scraping support server 5 may generate additional analysis information, such as a report on the status of funds or a company's financial statements, based on the scraped user's deposit/withdrawal transaction details and transmit it to the user device 2.

Additionally, in one embodiment, the input data that the scraping support server 5 transmits to the user device 2 and/or the output data that the user device 2 transmits to the scraping support server 5 may be at least partially encrypted. there is. This is to prevent personal information of the scraped user from being leaked even if there is a malicious third party intercepting the communication between the scraping support server 5 and the user device 2. For example, input data and output data may be transmitted by encrypting pre-set important information based on the type and/or number of data items the user wishes to scrape or the identification information of the information provider for scraping. The encryption method is not limited to this.

Figure 5 is a conceptual diagram showing the structure of input data used in a scraping service method according to an embodiment.

Referring to FIG. 5, in the scraping service method according to this embodiment, input data 500 may include an authentication information unit 520 and a scraping request unit 530. The authentication information unit 520 may include commands for performing an identity authentication process of a user who is the information subject, for example, logging in or logging out, with respect to the information provider server. Additionally, the scraping request unit 530 is a part that defines the information to be collected from the information provider server. In one embodiment, the scraping request unit 530 may include a scraping command code 540 and one or more scraping request data 531, 532, ... 53n.

An example of the configuration of input data as shown in FIG. 5 in code form is shown in Table 1 below. In 1 below, the "COMMON_HEAD" part corresponds to the common part 510 of FIG. 5 and may include an API identification code by the scraping support server and an API code to be performed next.

{
"COMMON_HEAD" : {
"API_CD" : "API identification code",
"NEXT_API_CD" : "API code to perform next"
},
"REQ_DATA" : {
"CERT_INFO": [
"LOGIN_INFO" : {
Scraping login request data
},
"LOGOUT_INFO" : {
Scraping logout request data
}
],
"SCRP_REQ_INFO": {
"SCRP_CMD_CD": "Scraping command code",
"SCRP_REQ_LIST": [
{
Scraping Request Data 1
},
{
Scraping Request Data 2
},
{
Scraping request data n
}
]
}
}
}

Additionally, in Table 1 above, the “REQ_DATA” part corresponds to the authentication information unit 520. For example, the authentication information unit 520 may be a code of the type shown in Tables 2 and 3 below. Table 2 shows the authentication information unit 520 used when requesting to log in to the bank server, and Table 3 shows the authentication information unit 520 used when logging out from the bank server.

{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Login",
"Input": {
“Login method”: “CERT”,
"User ID": "",
"User Password": "",
"certification": {
"certification": {
"Name": "cn=Hong Gil-dong()0004040902779989,ou=KMB,ou=personal4IB,o=yessign,c=kr",
"Expiration date": "20171210",
"password": "********"
}
}
}

{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Logout",
"Input": {}
}

Additionally, in Table 1, the “SCRP_REQ_INFO” part corresponds to the scraping request unit 530. Among these, “SCRP_REQ_CD” corresponds to the scraping command code 540 and defines the command rule under which scraping will be performed on one or more request data defined through input data. For example, scraping for multiple requests is executed simultaneously, sequentially, or sequentially depending on which of the preset values (e.g., 1, 2, 3, etc.) of the "SCRP_REQ_CD" part is. If an error occurs, the scraping job can be configured to abort without processing the next request.

Furthermore, in Table 1 above, the “SCRP_REQ_LIST” part corresponds to one or more scraping request data (531, 532, ... 53n). For example, the scraping request data (531, 532, ... 53n) may be a code as shown in Table 4 below, and Table 4 shows the scraping request used when requesting an account inquiry from a bank server. Represents data.

{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Logout",
"Input": {}
}

FIG. 6 is a conceptual diagram showing the structure of output data used in the scraping service method according to an embodiment. In this embodiment, the output data may have a structure corresponding to the structure of the input data described above with reference to FIG. 5.

Referring to FIG. 6, in the scraping service method according to this embodiment, output data 600 may include an authentication information unit 620 and a scraping response unit 630. The authentication information unit 620 is a part that displays the identity authentication result of the user as the information subject, for example, the log-in or log-out result, with respect to the information provision agency server. Additionally, the scraping response unit 630 is a part that defines the scraping results provided by the information provision agency server. In one embodiment, the scraping response unit 630 may include a scraping command code 640 and one or more scraping result data 631, 632, ... 63n.

An example of the configuration of output data as shown in FIG. 6 in code form is shown in Table 5 below. In 1 below, the "COMMON_HEAD" part corresponds to the common part 610 of FIG. 6 and may include an API identification code by the scraping support server and an API code to be performed next.

{
"COMMON_HEAD" : {
"API_CD" : "API identification code",
"NEXT_API_CD" : "Next API code to request"
},
"CERT_INFO": [
"LOGIN_INFO" : {
Scraping login response data
},
"LOGOUT_INFO" : {
Scraping logout response data
}
],
"SCRP_RES_INFO": {
"SCRP_CMD_CD": "Scraping command code",
"SCRP_RES_LIST": [
{
Scraping response data 1
},
{
Scraping response data 2
},
{
scraping response data n
}
]
}
}

Additionally, in Table 5 above, the “REQ_DATA” part corresponds to the authentication result unit 620. For example, the authentication result unit 620 may be a code of the type shown in Tables 6 and 7 below. Table 6 shows the user login response to the bank server, and Table 7 shows the response when the user logs out from the bank server.

{
"Module": "OOObank ",
"Class": "Corporate Banking",
"Job": "Login",
"Input": {
“Login method”: “CERT”,
"User ID": "",
"User Password": "",
"certification": {
"Name": "cn=Hong Gil-dong()0004040902779989,ou=KMB,ou=personal4IB,o=yessign,c=kr",
"Expiration date": "20171210",
"password": "********"
}
},
"Output": {
"ErrorCode": "00000000",
"ErrorMessage": "",
"Result": {}
}
}

{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Logout",
"Input": {},
"Output": {
"ErrorCode": "00000000",
"ErrorMessage": "",
"Result": {}
}

Additionally, in Table 5 above, the “SCRP_REQ_INFO” part corresponds to the scraping response unit 630. Among these, “SCRP_REQ_CD” corresponds to the scraping command code 640 and represents the command data delivered when scraping is requested through input data.

Furthermore, in Table 5 above, the “SCRP_REQ_LIST” part corresponds to one or more scraping result data (631, 632, ... 63n). For example, the scraping result data (631, 632, ... 63n) may be a code as shown in Table 8 below, and Table 8 shows the result data that the bank server responds to the account inquiry request. .

{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Pre-annual account inquiry",
"Input": {},
"Output": {
"ErrorCode": "00000000",
"ErrorMessage": "",
"Result": {
“Pre-annual account inquiry”: [
{
“Deposit name”: “Savings account”,
"Account number": "1234951045678",
"currency code": "",
"Balance": "17000",
“New date”: “20160101”,
“Account number extension”: “”
}
]
}
}
}

In one embodiment, the input/output data provided by the scraping support server may further include organization information data. Organization information data is data generated by verifying whether it is possible to scrape information from the relevant information providing organization, and may include, for example, an organization code, an organization status code, an organization status message, etc. At this time, the organization code refers to the identifier of each information providing organization. Additionally, the institution status code indicates whether information can be collected from the institution, such as whether scraping is possible or whether there is an institution failure. Furthermore, the institution status message refers to text data that defines the status of each institution in more detail.

In the input/output data configuration described above with reference to FIGS. 5 and 6, there may be a plurality of scraping request data and scraping result data. For example, the entire input data for simultaneous inquiry of deposit/withdrawal accounts and loan accounts from the information provider server may be configured as shown in Table 9 below.

{
"COMMON_HEAD": {
"API_CD": "SCMS_001",
"NEXT_API_CD": ""
},
"REQ_DATA": {
"CERT_INFO": [
"LOGIN_INFO": {
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Login",
"Input": {
“Login method”: “CERT”,
"User ID": "",
"User Password": "",
"certification": {
"Name": "cn=Hong Gil-dong()0004040902779989,ou=KMB,ou=personal4IB,o=yessign,c=kr",
"Expiration date": "20171210",
"password": "********"

}
}
},
"LOGOUT_INFO": {
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Logout",
"Input": {

}
}
],
"SCRP_REQ_INFO": {
"SCRP_CMD_CD": "1",
"SCRP_REQ_LIST": [
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Pre-annual account inquiry",
"Input": {
“Administrator Password”:”1234”
}
},
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Loan account inquiry",
"Input": {

}
}
]
}
}
}

In embodiments of the present invention, the scraping engine mounted on the user device automatically performs user login by transmitting such input data to the information provider server and collects information (e.g., account transaction details) provided by the information provider server. etc.) can be received. At this time, the format and structure of information required for input data may be changed according to the communication policy of the information provider server.

For example, when the request data for checking deposit/withdrawal accounts and loan accounts using the scraping method is configured as shown in Table 10 below, the bank's communication policy has been changed to require more [administrator password] when checking deposit/withdrawal accounts. Assume the case.

"SCRP_REQ_LIST": [
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Pre-annual account inquiry",
"Input": {
}
},
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Loan account inquiry",
"Input": {

In this case, the scraping engine requires modified input data as shown in Table 11 below in order to normally scrape deposit/withdrawal account inquiry details. Conventionally, in order for the scraping engine to be able to generate such modified input data, the scraping engine application installed on the user device had to be updated and distributed.

However, according to embodiments of the present invention, when an information request specifying an information provider (e.g., bank) and type of information (e.g., account transaction details, etc.) is received from the user device, the scraping support server uses the server-side API. By changing the format and data, input data as shown in Table 11 can be generated. Therefore, even if the communication policy of the information provider changes, there is an advantage of being able to respond easily and quickly by modifying the API of the scraping support server without the need to modify the scraping engine or application.

"SCRP_REQ_LIST": [
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Pre-annual account inquiry",
"Input": {
“Administrator Password”:”1234”
}
},
{
"Module": "OOObank",
"Class": "Corporate Banking",
"Job": "Loan account inquiry",
"Input": {

Referring again to Figure 2, in one embodiment, the data processing unit 52 of the scraping support server instead of providing input data corresponding to every information request received from the user device 2, previously scraped and stored in the storage unit ( 53), input data for scraping pre-stored information or new information may be selectively provided. For example, the scraping support server 5 processes all the input and output data packets necessary for a plurality of user devices 2 to scrape information from a plurality of information provision agency servers 3. It places a computational burden on the system and has the potential to cause data backlog or processing delays.

In order to prevent this problem, the scraping support server 5 receives the output data scraped by the user device 2, extracts user information therefrom, stores it in the storage unit 53, and extracts the user information from the user device 2. If there is a request for information again, the information stored in the storage unit 53 may be provided instead of providing input data for previously scraped information. On the other hand, when the user requests the same information again after providing the information stored in the storage unit 53, when the user requests information update, or when a predetermined period of time has elapsed after the information is stored in the storage unit 53 In this case, as in the above-described embodiments, the data processing unit 52 of the scraping support server 5 may generate input data for scraping and provide it to the user device 2. Through this, the burden of having to generate input/output data each time in response to frequent inquiries of information that is not updated well or duplicate inquiries of the same information can be reduced.

As another example, when there are multiple information requests from the same user device 2 within a certain time, the data processing unit 52 of the scraping support server 5 determines the requested data through these multiple information requests and makes these requests. Input data can also be created to scrape data through one input data. That is, referring to FIG. 5, scraping request data 531, 532, ... 53n containing data items requested through API at separate times from the user device 2 in one input data 500. By generating the input data 500, it is possible to prevent the application on the user device 2 from scraping the information provider server 3 at an unnecessarily frequent frequency.

The operation of the scraping service method according to the embodiments described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. Computer-readable recording media on which programs for implementing operations by the scraping service method according to embodiments are recorded include all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention discussed above has been described with reference to the embodiments shown in the drawings, but these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

Claims (18)

a communication unit configured to receive an information request from a user device to scrape one or more information; and
In response to the information request, a data processing unit configured to generate input data for the user device to scrape the one or more information from the information provider server, based on communication policy information of the information provider server,
The Department of Communications,
A scraping support server further configured to transmit the input data to the user device.
According to paragraph 1,
The communication unit is further configured to receive output data from the information provider server corresponding to the input data from the user device,
The scraping support server wherein the data processing unit is further configured to extract the one or more pieces of information from the output data based on communication policy information of the information provider server.
According to paragraph 2,
The communication unit is further configured to transmit the input data to the user device through a preset application programming interface and receive the output data from the user device.
According to paragraph 2,
Further comprising a storage unit configured to store the one or more information extracted by the data processing unit,
The scraping support server wherein the communication unit is further configured to provide the one or more information stored in the storage unit to the user device.
According to paragraph 1,
The input data includes an information authentication unit for authenticating the information provider server, a scraping command code defining a scraping rule, and one or more scraping request data defining a data item to be scraped.
According to paragraph 2,
The output data includes an authentication result section indicating an authentication result for the information provider server, a scraping command code defining a scraping rule, and one or more scraping result data defining a scraping result of a data item.
Memory storing executable instructions;
A communication module configured to communicate with a user device; and
A processor communicatively connected to the memory and the communication module,
The processor executes the instruction,
Send an information request to scrape one or more pieces of information to a scraping support server,
Receive input data corresponding to the information request from the scraping support server,
A user device for scraping configured to receive output data from the information provider server in a scraping manner by transmitting the input data to the information provider server.
In clause 7,
The processor is further configured to transmit the output data to the scraping support server and receive the one or more information extracted from the output data from the scraping support server.
A scraping support server receiving an information request from a user device to scrape one or more information;
generating, by the scraping support server, input data for the user device to scrape the one or more pieces of information from the information providing agency server, based on communication policy information of the information providing agency server; and
A scraping service method comprising the step of transmitting, by the scraping support server, the input data to the user device.
According to clause 9,
The step of transmitting the input data to the user device includes the step of the scraping support server transmitting the input data to the user device through a preset application programming interface.
According to clause 9,
receiving, by the scraping support server, output data of the information provider server corresponding to the input data from the user device; and
The scraping service method further includes the step of extracting, by the scraping support server, the one or more pieces of information from the output data based on communication policy information of the information provider server.
According to clause 11,
The receiving of the output data includes the scraping support server receiving the output data from the user device through a preset application programming interface.
According to clause 11,
storing, by the scraping support server, the one or more pieces of information extracted from the output data; and
The scraping service method further comprising providing, by the scraping support server, the one or more information stored in the scraping support server to the user device.
According to clause 9,
The input data includes an information authentication unit for authenticating the information provider server, a scraping command code defining a scraping rule, and one or more scraping request data defining a data item to be scraped. A scraping service method.
According to clause 11,
The output data includes an authentication result portion indicating an authentication result for the information provider server, a scraping command code defining a scraping rule, and one or more scraping result data defining a scraping result of a data item. A scraping service method.
A user device transmitting an information request to scrape one or more information to a scraping support server;
receiving, by the user device, input data corresponding to the information request from the scraping support server;
transmitting, by the user device, the input data to an information provider server; and
A scraping service method comprising the step of receiving, by the user device, output data by scraping from the information provider server in response to the input data.
According to clause 16,
transmitting, by the user device, the output data to the scraping support server; and
The scraping service method further comprising receiving, by the user device, the one or more information extracted from the output data from the scraping support server.
A computer program stored in a computer-readable recording medium combined with hardware to execute the scraping service method according to any one of claims 9 to 17.

Images