WO2017166644A1 - Data acquisition method and system - Google Patents
Data acquisition method and system Download PDFInfo
- Publication number
- WO2017166644A1 WO2017166644A1 PCT/CN2016/096968 CN2016096968W WO2017166644A1 WO 2017166644 A1 WO2017166644 A1 WO 2017166644A1 CN 2016096968 W CN2016096968 W CN 2016096968W WO 2017166644 A1 WO2017166644 A1 WO 2017166644A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data collection
- data
- log information
- module
- service
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Definitions
- the present disclosure relates to the field of data processing, for example, to a data acquisition method and system.
- the log is an event record called log generated by the network device, system, and service program at runtime; each log records the description information of the date, time, user, and action.
- the log records the life cycle of the system. By checking the log, you can know the status of the system at a certain moment. By analyzing the logs and collecting useful data, you can get the user's usage information and access statistics for the service system. Provide evidence for optimization and prevention of cybersecurity issues.
- the embodiments of the present disclosure provide a data collection method and system, which are used to solve the problem that the data collected by the related data collection method is not rich enough for the user behavior, and can enrich the data center platform, and can provide users with more humanization according to the data collection result.
- Embodiments of the present disclosure provide a data collection method, including:
- the target data collection result is stored.
- the embodiment of the present disclosure provides a data collection system, including: a log information storage module, a data collection task configuration module, a log data collection module, and a target data storage module;
- the log information storage module is configured to store log information of at least one service in a distributed storage manner
- the data collection task configuration module is configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
- the log data collection module is configured to collect target data from the log information stored in the log information storage module according to the data collection task rule;
- the target data storage module is configured to store the target data output by the log data collection module.
- Embodiments of the present disclosure provide a non-transitory computer readable storage medium storing computer executable instructions that, when executed by an electronic device, cause the electronic device to perform the data collection method described above.
- Embodiments of the present disclosure provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer Causing the computer to perform the data acquisition method described above.
- An embodiment of the present disclosure provides an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor, the instructions being The at least one processor, when executed, causes the at least one processor to perform the data acquisition method described above.
- the embodiments of the present disclosure provide a data collection method and system.
- storing log information in a distributed storage manner can effectively reduce the performance requirements of a single CPU and resources, and reduce the cost of data collection;
- the disclosed embodiment can be configured by parsing log information
- the data collection task rule collects the target data from the analysis result. Since a large amount of log information contains a large amount of user data, user operation behavior data and business data, the collection result not only enriches the data center platform, but also enables The data collection results effectively analyze the user's behavior habits and business operations to better understand the user's needs, thus providing users with more humanized services and better improving the business platform.
- FIG. 1 is a flow chart of steps of a first embodiment of a data collection method according to the present disclosure
- FIG. 2 is a schematic diagram of a user activity map of the present disclosure
- FIG. 3 is a flow chart of steps of a second embodiment of a data collection method according to the present disclosure.
- Embodiment 4 is a schematic structural diagram of Embodiment 1 of a data collection system according to the present disclosure
- FIG. 5 is a schematic structural diagram of Embodiment 2 of a data collection system according to the present disclosure.
- FIG. 6 is a schematic structural diagram of Embodiment 3 of a data collection system according to the present disclosure.
- FIG. 7 is a schematic diagram showing the hardware structure of an electronic device for data acquisition according to the present disclosure.
- FIG. 1 a flow chart of steps of a first embodiment of a data collection method of the present disclosure is shown. include:
- Step 101 Store log information of at least one service in a distributed storage manner.
- the log information of the service in the embodiment of the disclosure may be stored in a distributed storage manner, that is, the log file is evenly distributed to multiple data servers for storage, and the resources of the data servers are uniformly managed and allocated to the user.
- a file system access interface is provided, and a large number of log information files that can be effectively solved by the distributed storage method occupy too many resources, resulting in high requirements for multiple storage resources and CPU performance, and can effectively solve log information.
- the log information may include the user behavior habits and the service information data in the embodiment of the disclosure.
- the downloading time, the environment initialization time, the source detection time, and the source detection time are included in the transcoded log information.
- the media file log information (MediaInfo file) stored in the distributed file system may include: Data information such as code rate, frame rate, size, video duration, audio duration, video format, audio format, number of channels, video bit rate, audio bit rate, and the like.
- the above-mentioned transcoding log information and the media file log information are only an example of the log information of the embodiment of the present disclosure, and are not to be understood as the limitation of the log information in the embodiment of the present disclosure.
- the log information may include the user.
- the user-related log information generated by the operation, and the service-related log information generated during the service processing process, the log information is not limited in the embodiment of the present disclosure.
- the process of performing the distributed storage of the log information may refer to the process of the related distributed storage file information, which is not limited in the embodiment of the present disclosure.
- Step 102 Configure a data collection task rule according to the corresponding service requirement, where the data collection task rule corresponds to at least one feature information;
- the data collection task rule may be pre-defined according to requirements, and the corresponding data collection task rule may be formulated according to the pre-acquired target data and the feature information corresponding to the target data, because the pre-acquired target data may involve one or more Feature information, so the corresponding data collection task rule corresponds to one or more feature information, for example, the target data is the total number of videos successfully uploaded by the Shanghai node within the last month, and the feature information involved includes: regional characteristics The information is: Shanghai, the time range characteristic information is: the most recent month (for example: 2015/10/01 to 2015/11/01), the video upload status characteristic information is: three characteristics information are successfully obtained, and the data collection task rules are corresponding of Corresponding to the three feature information, the corresponding data collection task rule can be: the video with the geographical feature information is Shanghai, and the video upload time is between 2015/10/01 and 2015/11/01, and the video upload status is successful. quantity;
- the user may configure the data collection task rule on the user interaction interface.
- the service needs to count the total amount of videos successfully uploaded by the Shanghai node within the last month, that is, the target data is the latest one.
- the total number of videos successfully uploaded by the Shanghai node video is configured on the user interaction interface.
- the above data collection task rule is the statistical geographical feature information for Shanghai, and the video upload time is from 2015/10/01 to 2015/11/ 01, and the video upload status is the number of successful videos, correspondingly, you can configure the feature information: the regional feature information is: Shanghai, the time range feature information is: the most recent month (for example: 2015/10/01 to 2015/11 /01), the video upload status feature information is: success.
- the configuration of the data collection task rule by using the user interaction interface is only one way of configuring the data collection task rule in the embodiment of the present disclosure, and is not understood as a kind of configuration data collection task rule in the embodiment of the present disclosure.
- the configuration of the data collection task rule is not limited by the configuration of the data collection task rule in the embodiment of the present disclosure.
- Step 103 Collect target data from the stored log information according to the data collection task rule.
- the step 103 of collecting the target data from the stored log information according to the data collection task rule may include:
- Step A1 Obtain target log information of the corresponding service from the stored log information.
- Step A2 Parse the target log information to collect the target data from the parsing result according to the data collection task rule. due to
- the log information storage server may store log information of multiple services, such as an upload task, a download task, a storage task, and a transcoding task, etc., therefore, the empty page extracts the target of the corresponding service according to the service type.
- Log information for example, if the service type is uploaded, the log information corresponding to the upload task in the log information storage server is the target log information.
- the parsing result obtained by parsing the log information is: upload date: 2015-10-20; upload time: 10:00:30; user: 1001; IP: 10.80.25.32 Upload status: success
- collect the target data from the parsing result according to the data collection task rule that is, analyze the parsing result according to the data task collection rule to collect the target data
- the data collection task rule is a statistical regional feature.
- the number of videos whose video upload date is from 2015/10/01 to 2015/11/01 and the video upload status is successful, you can perform traversal analysis on the analysis result of the obtained log information, and the statistics meet the IP.
- the address corresponds to Shanghai, and the upload date is between 2015/10/01 and 2015/11/01, and the number of log messages with the status of success is the result of collecting the final target data.
- the foregoing data collection task rule is that the number of videos whose statistical geographical feature information is Shanghai and the video upload date is between 2015/10/01 and 2015/11/01 and the video upload status is successful is only as an embodiment of the present disclosure.
- An example of the data collection task rule is not understood as a limitation on the data collection task rule in the embodiment of the present disclosure.
- the data collection task rule can be set by a person in the field according to business requirements, for example, a service. You need to collect the number of failed video uploads in user B.
- the corresponding data collection task rule is: the user feature information is user A, and the time range feature information is time period B, and the upload status is failed log information. Total; embodiments of the present disclosure do not define data collection task rules herein.
- Step 104 Store the target data collection result.
- the embodiment of the present disclosure may further include:
- Reading log information from the at least one service is Reading log information from the at least one service.
- the embodiment of the present disclosure may access the service log information storage medium of the service, and read the log information from the service log information storage medium; or receive the log information uploaded by the service through the API interface.
- the disclosed embodiment does not limit the manner in which the log information is acquired.
- data can be collected from the log information, and the collected result can cover the data of the user, the content, the production process, and the research and development index, and then analyze the data to obtain more detailed data analysis, such as Obtaining user activity map analysis, user refinement operation, and the like.
- FIG. 2 a schematic diagram of a user activity map in the embodiment of the present disclosure is shown, which uses an application according to the collected data for a certain user. All log information generated is collected and analyzed.
- a data collection method uses a distributed storage manner to store log information, which can effectively reduce performance requirements for a single CPU and resources, and reduce the cost of data collection;
- the embodiment of the present disclosure can collect the target data according to the configured data collection task rule by parsing the log information. Since a large amount of log information includes a large amount of user data, user operation behavior data, and service data, the collection result is collected. It not only enriches the data center platform, but also enables effective analysis of user behavior habits and business operations based on data collection results to better understand user needs, thereby providing users with more personalized services and better improving the service platform. .
- FIG. 3 a flow chart of the steps of the second embodiment of the data collection method of the present disclosure is shown, which may include:
- Step 301 Store log information of at least one service in a distributed storage manner.
- Step 302 Configure a data collection task rule according to the corresponding service requirement; the data collection task rule corresponds to at least one feature information; wherein the data collection task further includes: data collection interface information;
- Step 303 Collect target data from the stored log information according to the data collection task rule.
- Step 304 Collect target data from the database according to the data collection task rule;
- Step 305 Store the target data collection result.
- a step 304 is added in the embodiment of the present disclosure, in which the target data can be collected from a database, and/or a text file, that is, the data may be based on multiple data sources in the embodiment of the present disclosure. Data collection and data collection results are more abundant.
- the data collection interface information may include: a name of the data source, a data storage manner, and a data storage format, where the data storage manner may include: a storage type and a storage location of the data, and the storage type may include: a database Type, for example: My sql, Oracle, etc., text type, such as: txt, syslog, etc., log information, such as: web log, operating system log, etc.; for database type data, the data collection interface information can also indicate the database where Host IP address, Database name, user name, password; for text type data, the data collection interface information may also indicate the storage path of the file; the data storage format refers to the format of the data itself, and for the database data, the above database storage format includes the data to be read. Data table name, field name, primary key information; for the text type, the above database storage format mainly includes the file name, and keyword information.
- a database Type for example: My sql, Oracle, etc.
- text type such as: txt,
- data is collected from a database to illustrate:
- the business needs to collect the source data from the production database
- the data collection interface information may include: data source: production database, data storage type: My sql, data storage location: upload task table and other information, from the above
- the source data uploading data is collected in the uploading task table of the production database, and the data may include: file size, file name, uploading user, uploading client IP, starting uploading time, uploading completion time, storage node, and the like.
- the data source of the current data collection task may be determined according to the data collection interface information as a database, or a text file, or log information.
- the foregoing method may further include:
- the distributed storage mode is used to store text file information of at least one service.
- the data information and the text file stored in the database may also be stored in a distributed storage manner, so that the requirements for the performance of the CPU and resources of the single machine can be reduced, thereby saving the cost of data collection.
- Embodiment 1 of a data collection system of the present disclosure may include: a log information storage module 401, a data collection task configuration module 402, a log data collection module 403, and a target data storage module 404. ;
- the log information storage module 401 can be configured to store log information of at least one service in a distributed storage manner.
- the data collection task configuration module 402 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
- the log data collection module 403 may be configured to collect target data from the log information stored in the log information storage module 401 according to the data collection task rule;
- the target data storage module 304 may be configured to store the target data output by the log data collection module.
- Embodiment 1 of a data collection system may include: a log information storage module 501, a data collection task configuration module 502, a log data collection module 503, and a target data storage module 504. ;
- the log information storage module 501 can be configured to store log information of at least one service in a distributed storage manner.
- the data collection task configuration module 502 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
- the log data collection module 503 may be configured to collect target data from the log information stored in the log information storage module 501 according to the data collection task rule;
- the target data storage module 504 can be configured to store the target data output by the log data collection module.
- the log data collection module 503 may include: a log information obtaining submodule 4031 and a log information parsing submodule 5032;
- the log information obtaining sub-module 5031 may be configured to obtain target log information of the corresponding service from the log information storage module.
- the log information parsing sub-module 5032 may be configured to parse the target log information acquired by the log information obtaining sub-module to collect the target data from the parsing result according to the data collecting task rule.
- FIG. 6 a schematic structural diagram of a third embodiment of a data collection system according to the present disclosure is shown, which may include: a log information storage module 601, a data collection task configuration module 602, and a log data collection module. 603, a database data collection module 604, a text file data collection module 605, and a target data storage module 606,
- the log information storage module 601 can be configured to store log information of at least one service in a distributed storage manner.
- the data collection task configuration module 602 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
- the log data collection module 603 may be configured to collect target data from the log information stored in the log information storage module 601 according to the data collection task rule;
- the target data storage module 606 may be configured to store the target data output by the log data collection module
- the database data collection module 604 can be configured to collect target data from the database according to the data collection task rule
- the text file data collection module 605 may be configured to collect target data from the text file according to the data collection task rule
- the data storage module 606 is further configured to store the database data collection module 604 and the target data output by the text file data collection module 605.
- the embodiment of the present disclosure may further include:
- a database data storage module configured to store database data information of at least one service in a distributed storage manner
- the text file data storage module can be configured to store the text file information of at least one service in a distributed storage manner.
- the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.
- the embodiment of the present application provides a non-transitory computer storage medium, where the computer storage medium stores computer executable instructions for performing the data collection method in any of the embodiments of the present application.
- FIG. 7 is a schematic diagram showing the hardware structure of an electronic device for data acquisition according to the present disclosure, as shown in FIG.
- the electronic device includes:
- processors 710 and memory 720 one processor 710 is taken as an example in FIG.
- the apparatus for performing the data collection method may further include: an input device 730 and an output device 740.
- the processor 710, the memory 720, the input device 730, and the output device 740 may be connected by a bus or other means, as exemplified by a bus connection in FIG.
- the memory 720 is used as a non-transitory computer readable storage medium, and can be used for storing a non-transitory software program, a non-transitory computer executable program, and a module, such as a program instruction corresponding to the data collection method in the embodiment of the present application. / Module (for example, each module shown in Figures 4-6).
- the processor 710 executes various functional applications and data processing of the server by running non-transitory software programs, instructions, and modules stored in the memory 720, that is, implementing the data acquisition method of the above method embodiment.
- the memory 720 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the data collection device, and the like.
- memory 720 can include high speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device.
- memory 720 can optionally include memory remotely located relative to processor 710, which can be connected to the data collection device over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
- Input device 730 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the data collection device.
- the output device 740 can include a display device such as a display screen.
- the one or more modules are stored in the memory 720, and when executed by the one or more processors 710, perform the data collection method in any of the above method embodiments.
- the electronic device of the embodiment of the present application exists in various forms, including but not limited to:
- Mobile communication devices These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication.
- Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.
- Ultra-mobile personal computer equipment This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access.
- Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.
- Portable entertainment devices These devices can display and play multimedia content. Such devices include: audio, video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.
- the server consists of a processor, a hard disk, a memory, a system bus, etc.
- the server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.
- the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, ie may be located A place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of the embodiment. Those of ordinary skill in the art can understand and implement without deliberate labor.
- the method can be implemented by means of software plus a general hardware platform, and of course, also through hardware.
- the above technical solutions may be embodied in the form of software products in essence or in the form of software products, which may be stored in a computer readable storage medium such as a ROM/RAM or a disk. , an optical disk, etc., includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in the various embodiments or portions of the various embodiments described above.
- the present application solves the problem that the data collected by the related data collection method is not rich enough for the user behavior, can enrich the data center platform, and can provide users with more humanized services according to the data collection results, and further improve the service platform.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
A data acquisition method and system. The method comprises: storing log information about at least one service by means of distributed storage (101); configuring a data acquisition task rule according to corresponding service requirements, wherein the data acquisition task rule corresponds to information about at least one feature (102); acquiring target data from the stored log information according to the data acquisition task rule (103); and storing a target data acquisition result (104). A data centre platform can be enriched, and a more humanized service can be provided for a user according to a data acquisition result, and a service platform is further improved.
Description
本申请要求在2016年03月31日提交中国专利局、申请号为2016102028782、公开名称为“一种数据采集方法和系统”的中国专利申请的优先权,该申请的全部内容通过引用结合在本申请中。The present application claims priority to Chinese Patent Application No. 2016102028782, entitled "A Data Collection Method and System", filed on March 31, 2016, the entire contents of which is incorporated herein by reference. In the application.
本公开涉及数据处理领域,例如涉及一种数据采集方法和系统。The present disclosure relates to the field of data processing, for example, to a data acquisition method and system.
随着现代科学技术的发展,数据采集技术已渗透到各行各业以及各种技术领域中。With the development of modern science and technology, data acquisition technology has penetrated into all walks of life and various technical fields.
日志是网络设备、系统及服务程序等在运行时产生的一个叫log的事件记录;每一行日志都记载着日期、时间、使用者及动作等相关操作的描述信息。日志记录了系统的生命周期,通过查阅日志,可以了解到系统在某个时刻所处的状态;通过对日志的分析,收集有用的数据,可以得到用户的使用信息和访问统计,为服务系统的优化和网络安全问题预防等提供依据。The log is an event record called log generated by the network device, system, and service program at runtime; each log records the description information of the date, time, user, and action. The log records the life cycle of the system. By checking the log, you can know the status of the system at a certain moment. By analyzing the logs and collecting useful data, you can get the user's usage information and access statistics for the service system. Provide evidence for optimization and prevention of cybersecurity issues.
但是相关的数据采集方法仅可以从数据库和网络文件中进行结构化数据的采集,却忽略了对于涵盖大量用户行为数据的日志信息中非结构化数据的采集,因此采集的关于用户行为的数据不够丰富。However, related data collection methods can only collect structured data from databases and network files, but ignore the collection of unstructured data in log information covering a large amount of user behavior data, so the data collected about user behavior is not enough. rich.
发明内容Summary of the invention
本公开实施例提供一种数据采集方法和系统,用以解决相关数据采集方法采集的关于用户行为的数据不够丰富的问题,能够丰富数据中心平台,同时可以根据数据采集结果为用户提供更人性化的服务,并更加的完善业务平台。The embodiments of the present disclosure provide a data collection method and system, which are used to solve the problem that the data collected by the related data collection method is not rich enough for the user behavior, and can enrich the data center platform, and can provide users with more humanization according to the data collection result. The service and the more perfect business platform.
本公开实施例提供一种数据采集方法,包括:Embodiments of the present disclosure provide a data collection method, including:
采用分布式存储方式存储至少一个业务的日志信息;
Collecting log information of at least one service in a distributed storage manner;
根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;Configuring a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据;Collecting target data from the stored log information according to the data collection task rule;
存储所述目标数据采集结果。The target data collection result is stored.
本公开实施例提供一种数据采集系统,包括:日志信息存储模块,数据采集任务配置模块,日志数据采集模块,及目标数据存储模块;The embodiment of the present disclosure provides a data collection system, including: a log information storage module, a data collection task configuration module, a log data collection module, and a target data storage module;
其中,所述日志信息存储模块,配置为采用分布式存储方式存储至少一个业务的日志信息;The log information storage module is configured to store log information of at least one service in a distributed storage manner;
所述数据采集任务配置模块,配置为根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;The data collection task configuration module is configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
所述日志数据采集模块,配置为根据所述数据采集任务规则从所述日志信息存储模块中存储的所述日志信息中采集目标数据;及The log data collection module is configured to collect target data from the log information stored in the log information storage module according to the data collection task rule; and
所述目标数据存储模块,配置为存储所述日志数据采集模块输出的所述目标数据。The target data storage module is configured to store the target data output by the log data collection module.
本公开实施例提供一种非暂态计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令被电子设备执行时,使得所述电子设备执行上述的数据采集方法。Embodiments of the present disclosure provide a non-transitory computer readable storage medium storing computer executable instructions that, when executed by an electronic device, cause the electronic device to perform the data collection method described above.
本公开实施例提供一种计算机程序产品,所述计算机程序产品包括存储在非暂态性计算机可读存储介质上的计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行上述的数据采集方法。Embodiments of the present disclosure provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer Causing the computer to perform the data acquisition method described above.
本公开实施例提供一种电子设备,包括至少一个处理器和与所述至少一个处理器通信连接的存储器,所述存储器用于存储可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行时,使所述至少一个处理器执行上述的数据采集方法。An embodiment of the present disclosure provides an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor, the instructions being The at least one processor, when executed, causes the at least one processor to perform the data acquisition method described above.
本公开实施例提供一种数据采集方法和系统,一方面,采用分布式存储方式存储日志信息,可以有效的降低对单机CPU及资源的性能要求,降低了数据采集的成本;另一方面,本公开实施例可以通过解析日志信息,从而根据配置
的数据采集任务规则从解析结果中进行目标数据的采集,由于大量的日志信息中包含了大量的用户数据、用户操作行为数据及业务数据,因此采集结果不但丰富了数据中心平台,同时可以使得根据数据采集结果对用户行为习惯及业务操作进行有效的分析,以更了解用户的需求,从而能够为用户提供更人性化的服务,并更加的完善业务平台。The embodiments of the present disclosure provide a data collection method and system. On the one hand, storing log information in a distributed storage manner can effectively reduce the performance requirements of a single CPU and resources, and reduce the cost of data collection; The disclosed embodiment can be configured by parsing log information
The data collection task rule collects the target data from the analysis result. Since a large amount of log information contains a large amount of user data, user operation behavior data and business data, the collection result not only enriches the data center platform, but also enables The data collection results effectively analyze the user's behavior habits and business operations to better understand the user's needs, thus providing users with more humanized services and better improving the business platform.
附图概述BRIEF abstract
一个或多个实施例通过与之对应的附图中的图片进行示例性说明,这些示例性说明并不构成对实施例的限定,附图中具有相同参考数字标号的元件表示为类似的元件,除非有特别申明,附图中的图不构成比例限制。The one or more embodiments are exemplified by the accompanying drawings in the accompanying drawings, and FIG. The figures in the drawings do not constitute a scale limitation unless otherwise stated.
图1为本公开的一种数据采集方法实施例一的步骤流程图;1 is a flow chart of steps of a first embodiment of a data collection method according to the present disclosure;
图2为本公开的一种用户活动图谱示意图;2 is a schematic diagram of a user activity map of the present disclosure;
图3为本公开的一种数据采集方法实施例二的步骤流程图;3 is a flow chart of steps of a second embodiment of a data collection method according to the present disclosure;
图4为本公开的一种数据采集系统实施例一的结构示意图;4 is a schematic structural diagram of Embodiment 1 of a data collection system according to the present disclosure;
图5为本公开的一种数据采集系统实施例二的结构示意图;FIG. 5 is a schematic structural diagram of Embodiment 2 of a data collection system according to the present disclosure;
图6为本公开的一种数据采集系统实施例三的结构示意图;以及FIG. 6 is a schematic structural diagram of Embodiment 3 of a data collection system according to the present disclosure;
图7为本公开的一种数据采集的电子设备的硬件结构示意图。FIG. 7 is a schematic diagram showing the hardware structure of an electronic device for data acquisition according to the present disclosure.
为使本公开实施例的技术方案和优点更加清楚,下面将结合本公开实施例中的附图,对本公开实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本公开一部分实施例,而不是全部的实施例。基于本公开中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本公开保护的范围。The technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the drawings in the embodiments of the present disclosure. Some embodiments are disclosed, rather than all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present disclosure without departing from the inventive scope are the scope of the disclosure.
方法实施例一Method embodiment 1
参照图1,示出了本公开的一种数据采集方法实施例一的步骤流程图,可以
包括:Referring to FIG. 1 , a flow chart of steps of a first embodiment of a data collection method of the present disclosure is shown.
include:
步骤101、采用分布式存储方式存储至少一个业务的日志信息;Step 101: Store log information of at least one service in a distributed storage manner.
本公开实施例中业务的日志信息可以采用分布式的存储方式进行存储,也即将日志文件均匀分布到多个数据服务器上进行存储,对这些数据服务器的资源进行统一的管理与分配,并向用户提供文件系统访问接口,采用分布式存储的方式可以有效解决的大量的日志信息文件占用的资源过多,导致对单机存储资源及CPU性能等多方面的要求较高的问题,可以有效解决日志信息文件大小、日志信息文件数量、打开日志信息文件数等的限制问题。The log information of the service in the embodiment of the disclosure may be stored in a distributed storage manner, that is, the log file is evenly distributed to multiple data servers for storage, and the resources of the data servers are uniformly managed and allocated to the user. A file system access interface is provided, and a large number of log information files that can be effectively solved by the distributed storage method occupy too many resources, resulting in high requirements for multiple storage resources and CPU performance, and can effectively solve log information. Limitations on file size, number of log information files, number of open log information files, etc.
本公开实施例中,日志信息可以包括本公开实施例中,日志信息可以包括用户行为习惯以及业务信息数据,例如,从转码日志信息中包括开始下载时间、环境初始化时间、片源检测时间、关键帧扫描时间、切片时间、文件转码时间、音频处理、字幕处理时间、成品检测时间等数据;再如:存储在分布式文件系统中的媒体文件日志信息(MediaInfo文件)中,可以包括:码率、帧率、大小、视频时长、音频时长、视频格式、音频格式、声道数、视频码率、音频码率等数据信息。In the embodiment of the present disclosure, the log information may include the user behavior habits and the service information data in the embodiment of the disclosure. For example, the downloading time, the environment initialization time, the source detection time, and the source detection time are included in the transcoded log information. The key frame scan time, the slicing time, the file transcoding time, the audio processing, the subtitle processing time, the finished product detection time, and the like. For example, the media file log information (MediaInfo file) stored in the distributed file system may include: Data information such as code rate, frame rate, size, video duration, audio duration, video format, audio format, number of channels, video bit rate, audio bit rate, and the like.
可以理解,上述转码日志信息以及媒体文件日志信息仅作为本公开实施例的日志信息的一种示例,而不理解为本公开实施例中日志信息的限定,实际上,日志信息可以包括由用户操作产生的用户相关日志信息,以及业务处理过程中产生的业务相关的日志信息,本公开实施例对日志信息不做限定。It can be understood that the above-mentioned transcoding log information and the media file log information are only an example of the log information of the embodiment of the present disclosure, and are not to be understood as the limitation of the log information in the embodiment of the present disclosure. In fact, the log information may include the user. The user-related log information generated by the operation, and the service-related log information generated during the service processing process, the log information is not limited in the embodiment of the present disclosure.
本公开实施例中,对日志信息进行分布式存储的过程参照相关分布式存储文件信息的过程即可,本公开实施例对此不作限制。In the embodiment of the present disclosure, the process of performing the distributed storage of the log information may refer to the process of the related distributed storage file information, which is not limited in the embodiment of the present disclosure.
步骤102、根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;Step 102: Configure a data collection task rule according to the corresponding service requirement, where the data collection task rule corresponds to at least one feature information;
本公开实施例中,上述数据采集任务规则可以根据需求预先制定,可以根据预采集的目标数据及目标数据对应的特征信息制定相应的数据采集任务规则,由于预采集的目标数据可能涉及一个或者多个特征信息,因此相应的数据采集任务规则对应一个或者多个特征信息,例如:目标数据为最近一个月之内,上海结点视频上传成功的视频总量,其中涉及的特征信息包括:地域特征信息为:上海,时间范围特征信息为:最近一个月(例如:2015/10/01至2015/11/01),视频上传状态特征信息为:成功共三个特征信息,则数据采集任务规则相应的
对应该三个特征信息,对应的数据采集任务规则即可以为:统计地域特征信息为上海、且视频上传时间在2015/10/01至2015/11/01间、且视频上传状态为成功的视频的数量;In the embodiment of the present disclosure, the data collection task rule may be pre-defined according to requirements, and the corresponding data collection task rule may be formulated according to the pre-acquired target data and the feature information corresponding to the target data, because the pre-acquired target data may involve one or more Feature information, so the corresponding data collection task rule corresponds to one or more feature information, for example, the target data is the total number of videos successfully uploaded by the Shanghai node within the last month, and the feature information involved includes: regional characteristics The information is: Shanghai, the time range characteristic information is: the most recent month (for example: 2015/10/01 to 2015/11/01), the video upload status characteristic information is: three characteristics information are successfully obtained, and the data collection task rules are corresponding of
Corresponding to the three feature information, the corresponding data collection task rule can be: the video with the geographical feature information is Shanghai, and the video upload time is between 2015/10/01 and 2015/11/01, and the video upload status is successful. quantity;
本公开实施例中,用户可以在用户交互界面上进行数据采集任务规则的配置,例如:业务需要统计最近一个月之内,上海结点视频上传成功的视频总量,也即目标数据为最近一个月之内,上海结点视频上传成功的视频总量,则在用户交互界面上配置上述数据采集任务规则为统计地域特征信息为上海、且视频上传时间在2015/10/01至2015/11/01间、且视频上传状态为成功的视频的数量,相应的,可以配置特征信息:地域特征信息为:上海,时间范围特征信息为:最近一个月(例如:2015/10/01至2015/11/01),视频上传状态特征信息为:成功。In the embodiment of the present disclosure, the user may configure the data collection task rule on the user interaction interface. For example, the service needs to count the total amount of videos successfully uploaded by the Shanghai node within the last month, that is, the target data is the latest one. Within the month, the total number of videos successfully uploaded by the Shanghai node video is configured on the user interaction interface. The above data collection task rule is the statistical geographical feature information for Shanghai, and the video upload time is from 2015/10/01 to 2015/11/ 01, and the video upload status is the number of successful videos, correspondingly, you can configure the feature information: the regional feature information is: Shanghai, the time range feature information is: the most recent month (for example: 2015/10/01 to 2015/11 /01), the video upload status feature information is: success.
可以理解,上述通过用户交互界面进行数据采集任务规则的配置仅作为本公开实施例中配置数据采集任务规则的一种方式,而不理解为是对本公开实施例中配置数据采集任务规则的一种限定,实际上也可以由本领域技术人员直接根据业务需求编写配置文件,从而实现数据采集任务规则的配置;本公开实施例在此对上述数据采集任务规则的配置方式不做限定。It can be understood that the configuration of the data collection task rule by using the user interaction interface is only one way of configuring the data collection task rule in the embodiment of the present disclosure, and is not understood as a kind of configuration data collection task rule in the embodiment of the present disclosure. The configuration of the data collection task rule is not limited by the configuration of the data collection task rule in the embodiment of the present disclosure.
步骤103、根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据;Step 103: Collect target data from the stored log information according to the data collection task rule.
在本公开的一种可选实施例中,所述根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据的步骤103,可以包括:In an optional embodiment of the present disclosure, the step 103 of collecting the target data from the stored log information according to the data collection task rule may include:
步骤A1、从所述存储的日志信息中获取对应业务的目标日志信息;Step A1: Obtain target log information of the corresponding service from the stored log information.
步骤A2、对所述目标日志信息进行解析,以根据所述数据采集任务规则从解析结果中采集所述目标数据。由于Step A2: Parse the target log information to collect the target data from the parsing result according to the data collection task rule. due to
本公开实施例中,日志信息存储服务器中可以存储多个业务的日志信息,例如:上传任务、下载任务、存储任务及转码任务等等,因此,空页根据业务种类来提取对应业务的目标日志信息,例如:业务类型为上传,则日志信息存储服务器中上传任务对应的日志信息即为目标日志信息。In the embodiment of the present disclosure, the log information storage server may store log information of multiple services, such as an upload task, a download task, a storage task, and a transcoding task, etc., therefore, the empty page extracts the target of the corresponding service according to the service type. Log information, for example, if the service type is uploaded, the log information corresponding to the upload task in the log information storage server is the target log information.
本公开实施例中,可以通过对日志信息进行解析获得日志信息中携带的数据,例如:获取的一条对应于上传业务的日志信息为:2015-10-20 10:00:30
user=1001upload a file IP=10.80.25.32 success,则解析该日志信息得到的解析结果为:上传日期:2015-10-20;上传时间:10:00:30;用户:1001;IP:10.80.25.32;上传状态:success;并根据数据采集任务规则从解析结果中采集所述目标数据,也即根据数据任务采集规则对解析结果进行分析,以采集目标数据,例如:数据采集任务规则为统计地域特征信息为上海、且视频上传日期在2015/10/01至2015/11/01间、且视频上传状态为成功的视频的数量,则可以对获取的日志信息的解析结果进行遍历分析,统计满足IP地址对应上海,且上传日期在2015/10/01至2015/11/01间,且上传状态为success的日志信息的数量,即为最终目标数据的采集结果。In the embodiment of the present disclosure, the log information is obtained by parsing the log information, for example, the obtained log information corresponding to the upload service is: 2015-10-20 10:00:30
User=1001upload a file IP=10.80.25.32 success, the parsing result obtained by parsing the log information is: upload date: 2015-10-20; upload time: 10:00:30; user: 1001; IP: 10.80.25.32 Upload status: success; and collect the target data from the parsing result according to the data collection task rule, that is, analyze the parsing result according to the data task collection rule to collect the target data, for example, the data collection task rule is a statistical regional feature. If the number of videos whose video upload date is from 2015/10/01 to 2015/11/01 and the video upload status is successful, you can perform traversal analysis on the analysis result of the obtained log information, and the statistics meet the IP. The address corresponds to Shanghai, and the upload date is between 2015/10/01 and 2015/11/01, and the number of log messages with the status of success is the result of collecting the final target data.
可以理解,上述数据采集任务规则为统计地域特征信息为上海、且视频上传日期在2015/10/01至2015/11/01间、且视频上传状态为成功的视频的数量仅作为本公开实施例中数据采集任务规则的一种实例,而不理解为是对本公开实施例中数据采集任务规则的一种限定,实际上,数据采集任务规则可以由本领域人员根据业务需求进行设定,例如:业务需要统计用户A在时间段B内上传失败的视频数量,则对应的数据采集任务规则为:统计用户特征信息为用户A,且时间范围特征信息为时间段B,且上传状态为失败的日志信息总数;本公开实施例在此不对数据采集任务规则进行限定。It can be understood that the foregoing data collection task rule is that the number of videos whose statistical geographical feature information is Shanghai and the video upload date is between 2015/10/01 and 2015/11/01 and the video upload status is successful is only as an embodiment of the present disclosure. An example of the data collection task rule is not understood as a limitation on the data collection task rule in the embodiment of the present disclosure. In fact, the data collection task rule can be set by a person in the field according to business requirements, for example, a service. You need to collect the number of failed video uploads in user B. The corresponding data collection task rule is: the user feature information is user A, and the time range feature information is time period B, and the upload status is failed log information. Total; embodiments of the present disclosure do not define data collection task rules herein.
步骤104、存储所述目标数据采集结果。Step 104: Store the target data collection result.
在本公开的一种可选实施例中,本公开实施例还可以包括:In an optional embodiment of the present disclosure, the embodiment of the present disclosure may further include:
接收至少一个业务上传的日志信息;和/或,Receiving log information uploaded by at least one service; and/or,
从所述至少一个业务中读取日志信息。Reading log information from the at least one service.
也即,本公开实施例可以通过访问业务的业务日志信息存储介质,并从业务日志信息存储介质中读取所述日志信息;也可以接收所述业务通过API接口上传的所述日志信息,本公开实施例在此对于日志信息的获取方式不做限定。That is, the embodiment of the present disclosure may access the service log information storage medium of the service, and read the log information from the service log information storage medium; or receive the log information uploaded by the service through the API interface. The disclosed embodiment does not limit the manner in which the log information is acquired.
本公开实施例中可以从日志信息中进行数据的采集,其采集结果中可以涵盖用户、内容、生产过程以及研发指标的数据,进而对上述数据进行分析,可以得到更为精细的数据分析,如得到用户活动图谱分析,用户精细化运营等等,例如:参照图2,示出了本公开实施例中一种用户活动图谱示意图,其尅根据对采集的数据为某一用户使用某一应用程序所产生的所有日志信息进行数据采集及分析得到。
In the embodiment of the present disclosure, data can be collected from the log information, and the collected result can cover the data of the user, the content, the production process, and the research and development index, and then analyze the data to obtain more detailed data analysis, such as Obtaining user activity map analysis, user refinement operation, and the like. For example, referring to FIG. 2, a schematic diagram of a user activity map in the embodiment of the present disclosure is shown, which uses an application according to the collected data for a certain user. All log information generated is collected and analyzed.
综上,本公开实施例中提供的一种数据采集方法,一方面,采用分布式存储方式存储日志信息,可以有效的降低对单机CPU及资源的性能要求,降低了数据采集的成本;另一方面,本公开实施例可以通过解析日志信息,从而根据配置的数据采集任务规则进行目标数据的采集,由于大量的日志信息中包含了大量的用户数据、用户操作行为数据及业务数据,因此采集结果不但丰富了数据中心平台,同时可以使得根据数据采集结果对用户行为习惯及业务操作进行有效的分析,以更了解用户的需求,从而能够为用户提供更人性化的服务,并更加的完善业务平台。In summary, a data collection method provided in an embodiment of the present disclosure, on the one hand, uses a distributed storage manner to store log information, which can effectively reduce performance requirements for a single CPU and resources, and reduce the cost of data collection; In an aspect, the embodiment of the present disclosure can collect the target data according to the configured data collection task rule by parsing the log information. Since a large amount of log information includes a large amount of user data, user operation behavior data, and service data, the collection result is collected. It not only enriches the data center platform, but also enables effective analysis of user behavior habits and business operations based on data collection results to better understand user needs, thereby providing users with more personalized services and better improving the service platform. .
方法实施例二Method embodiment two
参照图3,示出了本公开一种数据采集方法实施例二的步骤流程图,可以包括:Referring to FIG. 3, a flow chart of the steps of the second embodiment of the data collection method of the present disclosure is shown, which may include:
步骤301、采用分布式存储方式存储至少一个业务的日志信息;Step 301: Store log information of at least one service in a distributed storage manner.
步骤302、根据相应业务需求配置数据采集任务规则;所述数据采集任务规则对应至少一个特征信息;其中,所述数据采集任务中还包括:数据采集接口信息;Step 302: Configure a data collection task rule according to the corresponding service requirement; the data collection task rule corresponds to at least one feature information; wherein the data collection task further includes: data collection interface information;
步骤303、根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据;Step 303: Collect target data from the stored log information according to the data collection task rule.
步骤304、根据所述数据采集任务规则从所述数据库中采集目标数据;和/或Step 304: Collect target data from the database according to the data collection task rule; and/or
根据所述数据采集任务规则从所述文本文件中采集目标数据;Collecting target data from the text file according to the data collection task rule;
步骤305、存储所述目标数据采集结果。Step 305: Store the target data collection result.
相对于方法实施例一,本公开实施例中增加了步骤304,该步骤304中可以从数据库,和/或文本文件中进行目标数据的采集,也即本公开实施例中可以基于多个数据源进行数据的采集,数据采集结果更为丰富。With respect to the first embodiment of the present disclosure, a step 304 is added in the embodiment of the present disclosure, in which the target data can be collected from a database, and/or a text file, that is, the data may be based on multiple data sources in the embodiment of the present disclosure. Data collection and data collection results are more abundant.
本公开实施例中,上述数据采集接口信息可以包括:数据源的名称、数据存储方式和数据存储格式,其中,上述数据存储方式可以包括:数据的存储类型及存储位置,存储类型可以包括:数据库类型,例如:My sql、Oracle等,文本类型,如:txt、syslog等,日志信息,如:web日志,操作系统日志等;对于数据库类型的数据,数据采集接口信息中还可以指明数据库所在的主机IP地址、
数据库名称、用户名、密码;对于文本类型的数据,数据采集接口信息中还可以指明文件的存储路径;数据存储格式是指数据本身的格式,对于数据库数据,上述数据库存储格式包括要读取的数据表名,字段名,主键信息;对于文本类型,上述数据库存储格式则主要包括文件的名称,以及关键字信息。In the embodiment of the present disclosure, the data collection interface information may include: a name of the data source, a data storage manner, and a data storage format, where the data storage manner may include: a storage type and a storage location of the data, and the storage type may include: a database Type, for example: My sql, Oracle, etc., text type, such as: txt, syslog, etc., log information, such as: web log, operating system log, etc.; for database type data, the data collection interface information can also indicate the database where Host IP address,
Database name, user name, password; for text type data, the data collection interface information may also indicate the storage path of the file; the data storage format refers to the format of the data itself, and for the database data, the above database storage format includes the data to be read. Data table name, field name, primary key information; for the text type, the above database storage format mainly includes the file name, and keyword information.
以下通过示例对本公开实施例中,从数据库中采集数据加以说明:In the following, by way of example, in the embodiment of the present disclosure, data is collected from a database to illustrate:
例如,业务需要从生产数据库中采集片源上传数据,即数据采集接口信息中可以包括:数据源:生产数据库,数据存储类型::My sql,数据存储位置:上传任务表等信息,以从上述生产数据库的上传任务表中采集片源上传数据,所述数据可以包括:文件大小、文件名称、上传用户、上传客户端IP、开始上传时间、上传完成时间、存储节点等。For example, the business needs to collect the source data from the production database, that is, the data collection interface information may include: data source: production database, data storage type: My sql, data storage location: upload task table and other information, from the above The source data uploading data is collected in the uploading task table of the production database, and the data may include: file size, file name, uploading user, uploading client IP, starting uploading time, uploading completion time, storage node, and the like.
本公开实施例中可以根据数据采集接口信息确定当前数据采集任务的数据源是数据库,或者文本文件,或者日志信息。In the embodiment of the present disclosure, the data source of the current data collection task may be determined according to the data collection interface information as a database, or a text file, or log information.
在本公开的一种可选实施例中,上述方法还可以包括:In an optional embodiment of the disclosure, the foregoing method may further include:
采用分布式存储方式存储至少一个业务的数据库数据信息;和/或Storing database data information of at least one service in a distributed storage manner; and/or
采用分布式存储方式存储至少一个业务的文本文件信息。The distributed storage mode is used to store text file information of at least one service.
也即,本公开实施例中,数据库中存储的数据信息以及文本文件也可以采用分布式存储方式进行存储,以使得能够降低对单机的cpu及资源的性能的要求,进而节约数据采集的成本。That is, in the embodiment of the present disclosure, the data information and the text file stored in the database may also be stored in a distributed storage manner, so that the requirements for the performance of the CPU and resources of the single machine can be reduced, thereby saving the cost of data collection.
对于方法实施例,为了简单描述,故将其都表述为一系列的动作组合,但是本领域技术人员应该知悉,本申请实施例并不受所描述的动作顺序的限制,因为依据本申请实施例,某些步骤可以采用其他顺序或者同时进行。其次,本领域技术人员也应该知悉,说明书中所描述的实施例所涉及的动作并不一定是本申请实施例所必须的。For the method embodiments, for the sake of brevity, they are all described as a series of action combinations, but those skilled in the art should understand that the embodiments of the present application are not limited by the described action sequence, because the embodiment according to the present application Some steps can be performed in other orders or at the same time. Secondly, those skilled in the art should also understand that the actions involved in the embodiments described in the specification are not necessarily required in the embodiments of the present application.
装置实施例一Device embodiment 1
参照图4,示出了本公开的一种数据采集系统实施例一的结构示意图,可以包括:日志信息存储模块401,数据采集任务配置模块402,日志数据采集模块403,及目标数据存储模块404;Referring to FIG. 4, a schematic structural diagram of Embodiment 1 of a data collection system of the present disclosure is shown, which may include: a log information storage module 401, a data collection task configuration module 402, a log data collection module 403, and a target data storage module 404. ;
其中,所述日志信息存储模块401,可以配置为采用分布式存储方式存储至少一个业务的日志信息;
The log information storage module 401 can be configured to store log information of at least one service in a distributed storage manner.
所述数据采集任务配置模块402,可以配置为根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;The data collection task configuration module 402 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
所述日志数据采集模块403,可以配置为根据所述数据采集任务规则从所述日志信息存储模块401中存储的所述日志信息中采集目标数据;及The log data collection module 403 may be configured to collect target data from the log information stored in the log information storage module 401 according to the data collection task rule; and
所述目标数据存储模块304,可以配置为存储所述日志数据采集模块输出的所述目标数据。The target data storage module 304 may be configured to store the target data output by the log data collection module.
装置实施例二Device embodiment 2
参照图5,示出了本公开的一种数据采集系统实施例一的结构示意图,可以包括:日志信息存储模块501,数据采集任务配置模块502,日志数据采集模块503,及目标数据存储模块504;Referring to FIG. 5, a schematic structural diagram of Embodiment 1 of a data collection system according to the present disclosure is shown, which may include: a log information storage module 501, a data collection task configuration module 502, a log data collection module 503, and a target data storage module 504. ;
其中,所述日志信息存储模块501,可以配置为采用分布式存储方式存储至少一个业务的日志信息;The log information storage module 501 can be configured to store log information of at least one service in a distributed storage manner.
所述数据采集任务配置模块502,可以配置为根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;The data collection task configuration module 502 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
所述日志数据采集模块503,可以配置为根据所述数据采集任务规则从所述日志信息存储模块501中存储的所述日志信息中采集目标数据;及The log data collection module 503 may be configured to collect target data from the log information stored in the log information storage module 501 according to the data collection task rule; and
所述目标数据存储模块504,可以配置为存储所述日志数据采集模块输出的所述目标数据。The target data storage module 504 can be configured to store the target data output by the log data collection module.
其中,上述日志数据采集模块503,可以包括:日志信息获取子模块4031和日志信息解析子模块5032;其中,The log data collection module 503 may include: a log information obtaining submodule 4031 and a log information parsing submodule 5032;
所述日志信息获取子模块5031,可以配置为从所述日志信息存储模块中获取对应业务的目标日志信息;The log information obtaining sub-module 5031 may be configured to obtain target log information of the corresponding service from the log information storage module.
所述日志信息解析子模块5032,可以配置为对所述日志信息获取子模块获取的所述目标日志信息进行解析,以根据所述数据采集任务规则从解析结果中采集所述目标数据。The log information parsing sub-module 5032 may be configured to parse the target log information acquired by the log information obtaining sub-module to collect the target data from the parsing result according to the data collecting task rule.
装置实施例三Device embodiment three
参照图6,示出了本公开的一种数据采集系统实施例三的结构示意图,可以包括:日志信息存储模块601、数据采集任务配置模块602,日志数据采集模块
603,数据库数据采集模块604、文本文件数据采集模块605及目标数据存储模块606、Referring to FIG. 6, a schematic structural diagram of a third embodiment of a data collection system according to the present disclosure is shown, which may include: a log information storage module 601, a data collection task configuration module 602, and a log data collection module.
603, a database data collection module 604, a text file data collection module 605, and a target data storage module 606,
其中,所述日志信息存储模块601,可以配置为采用分布式存储方式存储至少一个业务的日志信息;The log information storage module 601 can be configured to store log information of at least one service in a distributed storage manner.
所述数据采集任务配置模块602,可以配置为根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;The data collection task configuration module 602 can be configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;
所述日志数据采集模块603,可以配置为根据所述数据采集任务规则从所述日志信息存储模块601中存储的所述日志信息中采集目标数据;The log data collection module 603 may be configured to collect target data from the log information stored in the log information storage module 601 according to the data collection task rule;
所述目标数据存储模块606,可以配置为存储所述日志数据采集模块输出的所述目标数据;The target data storage module 606 may be configured to store the target data output by the log data collection module;
所述数据库数据采集模块604,可以配置为根据所述数据采集任务规则从所述数据库中采集目标数据;及The database data collection module 604 can be configured to collect target data from the database according to the data collection task rule; and
所述文本文件数据采集模块605,可以配置为根据所述数据采集任务规则从所述文本文件中采集目标数据;The text file data collection module 605 may be configured to collect target data from the text file according to the data collection task rule;
则所述数据存储模块606,还可以配置为存储所述数据库数据采集模块604,及所述文本文件数据采集模块605输出的所述目标数据。The data storage module 606 is further configured to store the database data collection module 604 and the target data output by the text file data collection module 605.
在本公开的一种可选实施例中,本公开实施例中还可以包括:In an optional embodiment of the present disclosure, the embodiment of the present disclosure may further include:
数据库数据存储模块,可以配置为采用分布式存储方式存储至少一个业务的数据库数据信息;和/或a database data storage module, configured to store database data information of at least one service in a distributed storage manner; and/or
文本文件数据存储模块,可以配置为采用分布式存储方式存储至少一个业务的文本文件信息。The text file data storage module can be configured to store the text file information of at least one service in a distributed storage manner.
对于装置实施例而言,由于其与方法实施例基本相似,所以描述的比较简单,相关之处参见方法实施例的部分说明即可。For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.
本申请实施例提供了一种非暂态性计算机存储介质,所述计算机存储介质存储有计算机可执行指令,该计算机可执行指令用于执行本申请实施例中任一的数据采集方法。The embodiment of the present application provides a non-transitory computer storage medium, where the computer storage medium stores computer executable instructions for performing the data collection method in any of the embodiments of the present application.
图7是本公开的一种数据采集的电子设备的硬件结构示意图,如图7所示,
该电子设备包括:7 is a schematic diagram showing the hardware structure of an electronic device for data acquisition according to the present disclosure, as shown in FIG.
The electronic device includes:
一个或多个处理器710以及存储器720,图7中以一个处理器710为例。One or more processors 710 and memory 720, one processor 710 is taken as an example in FIG.
执行数据采集方法的设备还可以包括:输入装置730和输出装置740。The apparatus for performing the data collection method may further include: an input device 730 and an output device 740.
处理器710、存储器720、输入装置730和输出装置740可以通过总线或者其他方式连接,图7中以通过总线连接为例。The processor 710, the memory 720, the input device 730, and the output device 740 may be connected by a bus or other means, as exemplified by a bus connection in FIG.
存储器720作为一种非暂态性计算机可读存储介质,可用于存储非暂态性软件程序、非暂态性计算机可执行程序以及模块,如本申请实施例中的数据采集方法对应的程序指令/模块(例如,附图4-6中所示的各模块)。处理器710通过运行存储在存储器720中的非暂态性软件程序、指令以及模块,从而执行服务器的各种功能应用以及数据处理,即实现上述方法实施例数据采集方法。The memory 720 is used as a non-transitory computer readable storage medium, and can be used for storing a non-transitory software program, a non-transitory computer executable program, and a module, such as a program instruction corresponding to the data collection method in the embodiment of the present application. / Module (for example, each module shown in Figures 4-6). The processor 710 executes various functional applications and data processing of the server by running non-transitory software programs, instructions, and modules stored in the memory 720, that is, implementing the data acquisition method of the above method embodiment.
存储器720可以包括存储程序区和存储数据区,其中,存储程序区可存储操作系统、至少一个功能所需要的应用程序;存储数据区可存储根据数据采集装置的使用所创建的数据等。此外,存储器720可以包括高速随机存取存储器,还可以包括非暂态性存储器,例如至少一个磁盘存储器件、闪存器件、或其他非暂态性固态存储器件。在一些实施例中,存储器720可选包括相对于处理器710远程设置的存储器,这些远程存储器可以通过网络连接至数据采集装置。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。The memory 720 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the data collection device, and the like. Moreover, memory 720 can include high speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 720 can optionally include memory remotely located relative to processor 710, which can be connected to the data collection device over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
输入装置730可接收输入的数字或字符信息,以及产生与数据采集装置的用户设置以及功能控制有关的键信号输入。输出装置740可包括显示屏等显示设备。 Input device 730 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the data collection device. The output device 740 can include a display device such as a display screen.
所述一个或者多个模块存储在所述存储器720中,当被所述一个或者多个处理器710执行时,执行上述任意方法实施例中的数据采集方法。
The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, perform the data collection method in any of the above method embodiments.
上述产品可执行本申请实施例所提供的方法,具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节,可参见本申请实施例所提供的方法。The above products can perform the methods provided by the embodiments of the present application, and have the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.
本申请实施例的电子设备以多种形式存在,包括但不限于:The electronic device of the embodiment of the present application exists in various forms, including but not limited to:
(1)移动通信设备:这类设备的特点是具备移动通信功能,并且以提供话音、数据通信为主要目标。这类终端包括:智能手机(例如iPhone)、多媒体手机、功能性手机,以及低端手机等。(1) Mobile communication devices: These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.
(2)超移动个人计算机设备:这类设备属于个人计算机的范畴,有计算和处理功能,一般也具备移动上网特性。这类终端包括:PDA、MID和UMPC设备等,例如iPad。(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.
(3)便携式娱乐设备:这类设备可以显示和播放多媒体内容。该类设备包括:音频、视频播放器(例如iPod),掌上游戏机,电子书,以及智能玩具和便携式车载导航设备。(3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include: audio, video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.
(4)服务器:提供计算服务的设备,服务器的构成包括处理器、硬盘、内存、系统总线等,服务器和通用的计算机架构类似,但是由于需要提供高可靠的服务,因此在处理能力、稳定性、可靠性、安全性、可扩展性、可管理性等方面要求较高。(4) Server: A device that provides computing services. The server consists of a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.
(5)其他具有数据交互功能的电子设备。(5) Other electronic devices with data interaction functions.
以上所描述的装置实施例仅仅是示意性的,其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案。本领域普通技术人员在不付出创造性的劳动的情况下,即可以理解并实施。The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, ie may be located A place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of the embodiment. Those of ordinary skill in the art can understand and implement without deliberate labor.
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到各实施
方式可借助软件加通用硬件平台的方式来实现,当然也可以通过硬件。基于这样的理解,上述技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品可以存储在计算机可读存储介质中,如ROM/RAM、磁碟、光盘等,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)执行上述各个实施例或者实施例的某些部分所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand the implementations.
The method can be implemented by means of software plus a general hardware platform, and of course, also through hardware. Based on such understanding, the above technical solutions may be embodied in the form of software products in essence or in the form of software products, which may be stored in a computer readable storage medium such as a ROM/RAM or a disk. , an optical disk, etc., includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in the various embodiments or portions of the various embodiments described above.
以上实施例仅用以说明本公开的技术方案,而非对其限制;尽管参照前述实施例对本公开进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本公开各实施例技术方案的范围。The above embodiments are only used to illustrate the technical solutions of the present disclosure, and are not intended to be limiting; although the present disclosure has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that The technical solutions are described as being modified, or equivalent to some of the technical features are replaced; and such modifications or substitutions do not depart from the scope of the technical solutions of the embodiments of the present disclosure.
本申请解决了相关数据采集方法采集的关于用户行为的数据不够丰富的问题,能够丰富数据中心平台,同时可以根据数据采集结果为用户提供更人性化的服务,并更加的完善业务平台。
The present application solves the problem that the data collected by the related data collection method is not rich enough for the user behavior, can enrich the data center platform, and can provide users with more humanized services according to the data collection results, and further improve the service platform.
Claims (13)
- 一种数据采集方法,应用于电子设备,所述方法包括:A data collection method is applied to an electronic device, and the method includes:采用分布式存储方式存储至少一个业务的日志信息;Collecting log information of at least one service in a distributed storage manner;根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;Configuring a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据;以及Collecting target data from the stored log information according to the data collection task rule;存储所述目标数据采集结果。The target data collection result is stored.
- 根据权利要求1所述的方法,其中,所述根据所述数据采集任务规则从所述存储的所述日志信息中采集目标数据包括:The method of claim 1, wherein the collecting target data from the stored log information according to the data collection task rule comprises:从所述存储的日志信息里获取对应业务的目标日志信息;以及Obtaining target log information of the corresponding service from the stored log information;对所述目标日志信息进行解析,以根据所述数据采集任务规则从解析结果中采集所述目标数据。Parsing the target log information to collect the target data from the parsing result according to the data collection task rule.
- 根据权利要求1所述的方法,其中,所述数据采集任务中还包括:数据采集接口信息;则所述方法还包括:The method of claim 1, wherein the data collection task further comprises: data collection interface information; and the method further comprises:根据所述数据采集任务规则从所述数据库中采集目标数据;和/或Collecting target data from the database according to the data collection task rule; and/or根据所述数据采集任务规则从所述文本文件中采集目标数据。The target data is collected from the text file according to the data collection task rule.
- 根据权利要求3所述的方法,其中,所述方法还包括:The method of claim 3, wherein the method further comprises:采用分布式存储方式存储至少一个业务的数据库数据信息;和/或Storing database data information of at least one service in a distributed storage manner; and/or采用分布式存储方式存储至少一个业务的文本文件信息。The distributed storage mode is used to store text file information of at least one service.
- 根据权利要求1所述的方法,其中,所述方法还包括:The method of claim 1 wherein the method further comprises:接收至少一个业务上传的日志信息;和/或,Receiving log information uploaded by at least one service; and/or,从所述至少一个业务中读取日志信息。 Reading log information from the at least one service.
- 一种数据采集系统,其中,包括:日志信息存储模块,数据采集任务配置模块,日志数据采集模块,及目标数据存储模块;A data collection system, comprising: a log information storage module, a data collection task configuration module, a log data collection module, and a target data storage module;其中,所述日志信息存储模块,配置为采用分布式存储方式存储至少一个业务的日志信息;The log information storage module is configured to store log information of at least one service in a distributed storage manner;所述数据采集任务配置模块,配置为根据相应业务需求配置数据采集任务规则;其中,所述数据采集任务规则对应至少一个特征信息;The data collection task configuration module is configured to configure a data collection task rule according to the corresponding service requirement; wherein the data collection task rule corresponds to at least one feature information;所述日志数据采集模块,配置为根据所述数据采集任务规则从所述日志信息存储模块中存储的所述日志信息中采集目标数据;及The log data collection module is configured to collect target data from the log information stored in the log information storage module according to the data collection task rule; and所述目标数据存储模块,配置为存储所述日志数据采集模块输出的所述目标数据。The target data storage module is configured to store the target data output by the log data collection module.
- 根据权利要求6所述的系统,其中,所述日志数据采集模块,包括:日志信息获取子模块和日志信息解析子模块;其中,The system of claim 6, wherein the log data collection module comprises: a log information acquisition submodule and a log information parsing submodule;所述日志信息获取子模块,配置为从所述日志信息存储模块中获取对应业务的目标日志信息;The log information obtaining sub-module is configured to obtain target log information of the corresponding service from the log information storage module;所述日志信息解析子模块,配置为对所述日志信息获取子模块获取的所述目标日志信息进行解析,以根据所述数据采集任务规则从解析结果中采集所述目标数据。The log information parsing sub-module is configured to parse the target log information acquired by the log information obtaining sub-module to collect the target data from the parsing result according to the data collecting task rule.
- 根据权利要求6所述的系统,其中,所述数据采集任务中还包括:数据采集接口信息;则所述系统还包括:数据库数据采集模块,和/或文本文件数据采集模块;其中,The system of claim 6, wherein the data collection task further comprises: data collection interface information; and the system further comprises: a database data acquisition module, and/or a text file data acquisition module;所述数据库数据采集模块,配置为根据所述数据采集任务规则从所述数据库中采集目标数据;和/或The database data collection module is configured to collect target data from the database according to the data collection task rule; and/or所述文本文件数据采集模块,配置为根据所述数据采集任务规则从所述文 本文件中采集目标数据;The text file data collection module is configured to learn from the text according to the data collection task rule The target data is collected in this document;所述数据存储模块,还配置为存储所述数据库数据采集模块,和/或所述文本文件数据采集模块输出的所述目标数据。The data storage module is further configured to store the database data collection module, and/or the target data output by the text file data collection module.
- 根据权利要求8所述的系统,其中,所述系统还包括:The system of claim 8 wherein said system further comprises:数据库数据存储模块,配置为采用分布式存储方式存储至少一个业务的数据库数据信息;和/或a database data storage module configured to store database data information of at least one service in a distributed storage manner; and/or文本文件数据存储模块,配置为采用分布式存储方式存储至少一个业务的文本文件信息。The text file data storage module is configured to store the text file information of the at least one service in a distributed storage manner.
- 根据权利要求6所述的系统,其中,所述装置还包括:The system of claim 6 wherein said apparatus further comprises:接收日志信息模块,配置为接收至少一个业务上传的日志信息;和/或,Receiving a log information module, configured to receive log information uploaded by at least one service; and/or,读取日志信息模块,配置为从所述至少一个业务中读取日志信息。Reading a log information module, configured to read log information from the at least one service.
- 一种非暂态计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令被电子设备执行时,使得所述电子设备执行权利要求1-5任一项所述的方法。A non-transitory computer readable storage medium storing computer executable instructions that, when executed by an electronic device, cause the electronic device to perform the method of any of claims 1-5.
- 一种计算机程序产品,所述计算机程序产品包括存储在非暂态性计算机可读存储介质上的计算机程序,所述计算机程序包括程序指令,当所述程序指令被计算机执行时,使所述计算机执行权利要求1-5任一项所述的方法。A computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, causing the computer The method of any of claims 1-5.
- 一种电子设备,包括至少一个处理器和与所述至少一个处理器通信连接的存储器,所述存储器用于存储可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行时,使所述至少一个处理器执行权利要求1-5任一项所述的方法。 An electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor, the instructions being processed by the at least one When executed, the at least one processor is caused to perform the method of any of claims 1-5.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610202878.2A CN105912587A (en) | 2016-03-31 | 2016-03-31 | Data acquisition method and system |
CN201610202878.2 | 2016-03-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017166644A1 true WO2017166644A1 (en) | 2017-10-05 |
Family
ID=56745348
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/096968 WO2017166644A1 (en) | 2016-03-31 | 2016-08-26 | Data acquisition method and system |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN105912587A (en) |
WO (1) | WO2017166644A1 (en) |
Cited By (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109634929A (en) * | 2018-09-30 | 2019-04-16 | 阿里巴巴集团控股有限公司 | Acquisition method, device and the server of business datum |
CN109710605A (en) * | 2017-10-25 | 2019-05-03 | 卓望数码技术(深圳)有限公司 | A kind of facility information automation collection device and method |
CN110413496A (en) * | 2019-07-29 | 2019-11-05 | 福建南威软件有限公司 | A method of realizing that the operation data modularization of electronics license is collected |
CN110502514A (en) * | 2019-08-15 | 2019-11-26 | 中国平安财产保险股份有限公司 | Collecting method, device, equipment and computer readable storage medium |
CN110781248A (en) * | 2019-09-27 | 2020-02-11 | 浙江省北大信息技术高等研究院 | Multi-source heterogeneous data acquisition method and device |
CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
CN111343190A (en) * | 2020-03-05 | 2020-06-26 | 贵州宝智达网络科技有限公司 | Remote wireless data tamper-proof acquisition equipment and system |
CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
CN111611207A (en) * | 2020-05-21 | 2020-09-01 | 四川虹美智能科技有限公司 | State data processing method and device and computer equipment |
CN112214463A (en) * | 2019-07-12 | 2021-01-12 | 中科云谷科技有限公司 | Monitoring method, device and system of GPS terminal and storage medium |
CN112347180A (en) * | 2020-12-04 | 2021-02-09 | 航天信息股份有限公司企业服务分公司 | Data pushing method and electronic equipment |
CN112667728A (en) * | 2021-01-06 | 2021-04-16 | 上海振华重工(集团)股份有限公司 | Visual single-machine data acquisition method in wharf efficiency analysis |
CN113010240A (en) * | 2021-03-29 | 2021-06-22 | 北京金山云网络技术有限公司 | Data acquisition method, system, electronic equipment and storage medium |
CN113297298A (en) * | 2021-06-18 | 2021-08-24 | 中国农业银行股份有限公司 | Data statistical method, device, equipment and storage medium |
CN113377848A (en) * | 2020-02-25 | 2021-09-10 | 北京数聚鑫云信息技术有限公司 | Data processing method, device, equipment and storage medium |
CN113791946A (en) * | 2021-08-31 | 2021-12-14 | 北京达佳互联信息技术有限公司 | Log processing method and device, electronic equipment and storage medium |
CN114168509A (en) * | 2021-10-22 | 2022-03-11 | 中科苏州微电子产业技术研究院 | Expansion control method and system of data acquisition chip |
CN114189367A (en) * | 2021-11-30 | 2022-03-15 | 南京理工大学 | Safety log analysis system based on knowledge graph |
CN114328076A (en) * | 2021-09-18 | 2022-04-12 | 腾讯科技(深圳)有限公司 | Log information extraction method and device, computer equipment and storage medium |
CN114461490A (en) * | 2021-12-31 | 2022-05-10 | 广东航宇卫星科技有限公司 | Operation and maintenance data aggregation system |
CN114720761A (en) * | 2022-04-08 | 2022-07-08 | 北京汇能精电科技股份有限公司 | Configurable civil hybrid energy storage power supply data acquisition method and device |
CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术(深圳)股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
CN115102972A (en) * | 2022-07-15 | 2022-09-23 | 济南浪潮数据技术有限公司 | Method, device, equipment and medium for storing NFS (network file system) file |
WO2022205938A1 (en) * | 2021-03-30 | 2022-10-06 | 苏宁易购集团股份有限公司 | Data acquisition method and apparatus, computer device, and storage medium |
CN115278562A (en) * | 2022-06-24 | 2022-11-01 | 北京思特奇信息技术股份有限公司 | Method and system for managing and controlling short message reminding based on flow configuration, electronic device and storage medium |
CN116170299A (en) * | 2023-02-21 | 2023-05-26 | 上海洲邦信息科技有限公司 | Data acquisition method, system and device |
CN117061165A (en) * | 2023-08-10 | 2023-11-14 | 江苏瀚天智能科技股份有限公司 | Safety protection system based on space-time data lake technology of monitoring and control system |
CN117194179A (en) * | 2023-11-08 | 2023-12-08 | 杭州星锐网讯科技有限公司 | Index determination method and device, electronic equipment and storage medium |
CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
CN117290190A (en) * | 2023-11-27 | 2023-12-26 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
WO2024164709A1 (en) * | 2023-02-06 | 2024-08-15 | 天翼云科技有限公司 | Video user behavior data processing method, system, device and medium |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109327351B (en) * | 2018-09-12 | 2020-11-20 | 拉扎斯网络科技(上海)有限公司 | Method and device for collecting log data in real time, electronic equipment and storage medium |
CN109271531B (en) * | 2018-11-16 | 2023-04-18 | 苏州友教习亦教育科技有限公司 | Data management center based on operation and maintenance knowledge graph |
CN109918048B (en) * | 2018-12-27 | 2022-09-06 | 北京奇艺世纪科技有限公司 | Target object extraction method, device and system and computer readable storage medium |
CN110932918B (en) * | 2019-12-26 | 2023-01-10 | 远景智能国际私人投资有限公司 | Log data acquisition method and device and storage medium |
CN113126562B (en) * | 2020-01-16 | 2023-03-10 | 智能云科信息科技有限公司 | Data acquisition method, device and system and computer readable storage medium |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020178146A1 (en) * | 2001-05-24 | 2002-11-28 | International Business Machines Corporation | System and method for selective object history retention |
CN101610174A (en) * | 2009-07-24 | 2009-12-23 | 深圳市永达电子股份有限公司 | A kind of log correlation analysis system and method |
CN104869022A (en) * | 2015-05-27 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Log acquisition method and system |
CN104883365A (en) * | 2015-05-14 | 2015-09-02 | 浪潮电子信息产业股份有限公司 | Method and device for storing and reading security logs and security control system |
CN105099764A (en) * | 2015-06-29 | 2015-11-25 | 百度在线网络技术(北京)有限公司 | Log processing method and log processing device |
-
2016
- 2016-03-31 CN CN201610202878.2A patent/CN105912587A/en active Pending
- 2016-08-26 WO PCT/CN2016/096968 patent/WO2017166644A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020178146A1 (en) * | 2001-05-24 | 2002-11-28 | International Business Machines Corporation | System and method for selective object history retention |
CN101610174A (en) * | 2009-07-24 | 2009-12-23 | 深圳市永达电子股份有限公司 | A kind of log correlation analysis system and method |
CN104883365A (en) * | 2015-05-14 | 2015-09-02 | 浪潮电子信息产业股份有限公司 | Method and device for storing and reading security logs and security control system |
CN104869022A (en) * | 2015-05-27 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Log acquisition method and system |
CN105099764A (en) * | 2015-06-29 | 2015-11-25 | 百度在线网络技术(北京)有限公司 | Log processing method and log processing device |
Cited By (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109710605A (en) * | 2017-10-25 | 2019-05-03 | 卓望数码技术(深圳)有限公司 | A kind of facility information automation collection device and method |
CN109710605B (en) * | 2017-10-25 | 2022-12-16 | 卓望数码技术(深圳)有限公司 | Automatic equipment information acquisition device and method |
CN109634929A (en) * | 2018-09-30 | 2019-04-16 | 阿里巴巴集团控股有限公司 | Acquisition method, device and the server of business datum |
CN110968561A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Log storage method and distributed system |
CN110968561B (en) * | 2018-09-30 | 2024-02-13 | 北京国双科技有限公司 | Log storage method and distributed system |
CN112214463A (en) * | 2019-07-12 | 2021-01-12 | 中科云谷科技有限公司 | Monitoring method, device and system of GPS terminal and storage medium |
CN112214463B (en) * | 2019-07-12 | 2024-07-02 | 中科云谷科技有限公司 | GPS terminal monitoring method, device, system and storage medium |
CN110413496A (en) * | 2019-07-29 | 2019-11-05 | 福建南威软件有限公司 | A method of realizing that the operation data modularization of electronics license is collected |
CN110413496B (en) * | 2019-07-29 | 2022-08-19 | 福建南威软件有限公司 | Method for realizing componentized collection of electronic license operation data |
CN110502514A (en) * | 2019-08-15 | 2019-11-26 | 中国平安财产保险股份有限公司 | Collecting method, device, equipment and computer readable storage medium |
CN110502514B (en) * | 2019-08-15 | 2023-06-27 | 中国平安财产保险股份有限公司 | Data acquisition method, device, equipment and computer readable storage medium |
CN110781248A (en) * | 2019-09-27 | 2020-02-11 | 浙江省北大信息技术高等研究院 | Multi-source heterogeneous data acquisition method and device |
CN113377848A (en) * | 2020-02-25 | 2021-09-10 | 北京数聚鑫云信息技术有限公司 | Data processing method, device, equipment and storage medium |
CN111343190A (en) * | 2020-03-05 | 2020-06-26 | 贵州宝智达网络科技有限公司 | Remote wireless data tamper-proof acquisition equipment and system |
CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
CN111611207A (en) * | 2020-05-21 | 2020-09-01 | 四川虹美智能科技有限公司 | State data processing method and device and computer equipment |
CN111611207B (en) * | 2020-05-21 | 2023-06-23 | 四川虹美智能科技有限公司 | State data processing method and device and computer equipment |
CN112347180A (en) * | 2020-12-04 | 2021-02-09 | 航天信息股份有限公司企业服务分公司 | Data pushing method and electronic equipment |
CN112347180B (en) * | 2020-12-04 | 2023-08-01 | 航天信息股份有限公司企业服务分公司 | Data pushing method and electronic equipment |
CN112667728A (en) * | 2021-01-06 | 2021-04-16 | 上海振华重工(集团)股份有限公司 | Visual single-machine data acquisition method in wharf efficiency analysis |
CN112667728B (en) * | 2021-01-06 | 2023-11-21 | 上海振华重工(集团)股份有限公司 | Visual single machine data acquisition method in wharf efficiency analysis |
CN113010240B (en) * | 2021-03-29 | 2024-02-02 | 北京金山云网络技术有限公司 | Data acquisition method, system, electronic equipment and storage medium |
CN113010240A (en) * | 2021-03-29 | 2021-06-22 | 北京金山云网络技术有限公司 | Data acquisition method, system, electronic equipment and storage medium |
WO2022205938A1 (en) * | 2021-03-30 | 2022-10-06 | 苏宁易购集团股份有限公司 | Data acquisition method and apparatus, computer device, and storage medium |
CN113297298A (en) * | 2021-06-18 | 2021-08-24 | 中国农业银行股份有限公司 | Data statistical method, device, equipment and storage medium |
CN113791946A (en) * | 2021-08-31 | 2021-12-14 | 北京达佳互联信息技术有限公司 | Log processing method and device, electronic equipment and storage medium |
CN114328076B (en) * | 2021-09-18 | 2024-04-30 | 腾讯科技(深圳)有限公司 | Log information extraction method, device, computer equipment and storage medium |
CN114328076A (en) * | 2021-09-18 | 2022-04-12 | 腾讯科技(深圳)有限公司 | Log information extraction method and device, computer equipment and storage medium |
CN114168509A (en) * | 2021-10-22 | 2022-03-11 | 中科苏州微电子产业技术研究院 | Expansion control method and system of data acquisition chip |
CN114189367A (en) * | 2021-11-30 | 2022-03-15 | 南京理工大学 | Safety log analysis system based on knowledge graph |
CN114461490B (en) * | 2021-12-31 | 2023-05-30 | 广东航宇卫星科技有限公司 | Fortune dimension aggregation system |
CN114461490A (en) * | 2021-12-31 | 2022-05-10 | 广东航宇卫星科技有限公司 | Operation and maintenance data aggregation system |
CN114720761A (en) * | 2022-04-08 | 2022-07-08 | 北京汇能精电科技股份有限公司 | Configurable civil hybrid energy storage power supply data acquisition method and device |
CN115278562A (en) * | 2022-06-24 | 2022-11-01 | 北京思特奇信息技术股份有限公司 | Method and system for managing and controlling short message reminding based on flow configuration, electronic device and storage medium |
CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术(深圳)股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
CN115102972A (en) * | 2022-07-15 | 2022-09-23 | 济南浪潮数据技术有限公司 | Method, device, equipment and medium for storing NFS (network file system) file |
WO2024164709A1 (en) * | 2023-02-06 | 2024-08-15 | 天翼云科技有限公司 | Video user behavior data processing method, system, device and medium |
CN116170299A (en) * | 2023-02-21 | 2023-05-26 | 上海洲邦信息科技有限公司 | Data acquisition method, system and device |
CN117061165A (en) * | 2023-08-10 | 2023-11-14 | 江苏瀚天智能科技股份有限公司 | Safety protection system based on space-time data lake technology of monitoring and control system |
CN117194179B (en) * | 2023-11-08 | 2024-04-16 | 杭州星锐网讯科技有限公司 | Index determination method and device, electronic equipment and storage medium |
CN117194179A (en) * | 2023-11-08 | 2023-12-08 | 杭州星锐网讯科技有限公司 | Index determination method and device, electronic equipment and storage medium |
CN117251499B (en) * | 2023-11-15 | 2024-02-06 | 山东光合云谷大数据有限公司 | Data acquisition system |
CN117251499A (en) * | 2023-11-15 | 2023-12-19 | 山东光合云谷大数据有限公司 | Data acquisition system |
CN117290190B (en) * | 2023-11-27 | 2024-02-13 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
CN117290190A (en) * | 2023-11-27 | 2023-12-26 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN105912587A (en) | 2016-08-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2017166644A1 (en) | Data acquisition method and system | |
US10636425B2 (en) | Voice application platform | |
CN110943961B (en) | Data processing method, device and storage medium | |
US10733079B2 (en) | Systems and methods for end-to-end testing of applications using dynamically simulated data | |
US11196839B1 (en) | System and method for classifying API requests in API processing systems using a tree configuration | |
US9954746B2 (en) | Automatically generating service documentation based on actual usage | |
US10178156B2 (en) | Extraction and capture of information from customizable header | |
US9420068B1 (en) | Log streaming facilities for computing applications | |
CN105516821B (en) | The method and device of barrage screening | |
US9442967B2 (en) | Systems and methods for efficient data ingestion and query processing | |
US11775993B2 (en) | Generating customized surveys using third-party social networking information | |
WO2017177630A1 (en) | Method and device for recommending personalized information | |
US20130268516A1 (en) | Systems And Methods For Analyzing And Visualizing Social Events | |
US20150143215A1 (en) | Method and system for accessing audio/video community virtual rooms | |
US20170185454A1 (en) | Method and Electronic Device for Determining Resource Consumption of Task | |
US11437029B2 (en) | Voice application platform | |
WO2017113840A1 (en) | Information recommending method and device | |
US20110258154A1 (en) | Content duration and interaction monitoring to automate presentation of media content in a channel sharing of media content in a channel | |
KR102082063B1 (en) | How to Display Media Information, Servers, and Data Storage Media | |
US9582586B2 (en) | Massive rule-based classification engine | |
CN106230809B (en) | A kind of mobile Internet public sentiment monitoring method and system based on URL | |
CN104317804A (en) | Voting information publishing method and device | |
WO2017107831A1 (en) | Application based broadcasting method, device and electronic device | |
US11386152B1 (en) | Automatic generation of highlight clips for events | |
CN113051460A (en) | Elasticissearch-based data retrieval method and system, electronic device and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16896414 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16896414 Country of ref document: EP Kind code of ref document: A1 |