KR20190008673A - File indexing apparatus and method thereof using asynchronous method - Google Patents

Abstract

A file indexing apparatus using an asynchronous method and a method thereof are disclosed. The file indexing apparatus according to an embodiment of the present invention includes an index list generating part for generating an index list of files in a folder to be indexed, an index information extracting part for extracting the index information of the file from the generated index list, and an index information extracting part which uses the generated index list to determine whether the file needs to be indexed, and uses the index information of the file needing to be indexed to generate an index, and simultaneously performs index generation on another file which is subjected to index information extraction while the index information extracting part extracts the index information. It is possible to reduce an index time.

Description

[0001] File indexing apparatus and method using asynchronous method [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a file management technique, and more particularly, to a technique for improving processing speed and operability according to a plurality of files.

An index can be generated from a plurality of files recorded on a recording medium such as a disk and the file can be searched using the index. When you want to quickly search for desired contents in a large number of files, the process of indexing the files in the disk and converting them into a searchable index is called indexing, and the result of this process is referred to as an index do.

According to one embodiment, a file indexing apparatus and method for shortening the indexing time are proposed.

According to an embodiment of the present invention, there is provided a file indexing apparatus including an index list generating unit for generating an index list of files in a folder to be indexed, an index information extracting unit for extracting file index information from the generated index list, And an index is generated using the index information of the file that requires indexing. While the index information extracting unit extracts the index information, the indexing of another file in which the index information is extracted is simultaneously performed And an index processing unit for performing the processing.

The index list can include the file's last modified date and file path information.

The index list generating unit may include a thread pool including a plurality of threads for generating an index list on a folder basis and storing the generated index list on a folder basis. At this time, the index list generation unit manages the number of threads constituting the thread pool, and can enter the waiting state when the number of threads exceeds a predetermined number.

The index information extracting unit analyzes the file characteristics of the file to be indexed, determines which index information to extract, and extracts index information corresponding to the file characteristics. The index information extracting unit may include a producer thread pool including a plurality of producer threads for extracting the index information and providing the extracted index information to the index processing unit. At this time, the index information extracting unit manages the number of producer threads constituting the producer thread pool, and can enter the waiting state when the number of threads exceeds the preset number.

The index processor monitors a wait queue in which at least one index information extracted by the index information extractor is stored, and whether there is index information in the wait queue. If the index information exists, the index processor fetches the index information and performs index processing, And storing the generated index in the index DB.

The index processing unit creates a temporary folder, adds contents to be indexed in the temporary folder to the record, and if the number of additional records exceeds a preset number, the index processing unit calls the index query to perform a batch index. If the query execution is successful, The number of additional records can be initialized.

The index processor can separate the handle that is accessible to the table in the index DB into a read handle and a write handle, and index it using a handle that is appropriate for the purpose of the query.

According to one embodiment, an efficient indexing technique may be provided for application to a solution for searching files in a disc. In particular, the slowing down of each process associated with the index can improve the problem using asynchronous methods.

1 is a configuration diagram of a file indexing apparatus according to an embodiment of the present invention;
2 is a flowchart illustrating a file indexing method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. , Which may vary depending on the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

Each block of the accompanying block diagrams and combinations of steps of the flowcharts may be performed by computer program instructions (execution engines), which may be stored in a general-purpose computer, special purpose computer, or other processor of a programmable data processing apparatus The instructions that are executed through the processor of the computer or other programmable data processing equipment will generate means for performing the functions described in each block or flowchart of the block diagram.

These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce an article of manufacture containing instruction means for performing the functions described in each block or flowchart of the flowchart.

And computer program instructions may be loaded onto a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the data processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or step may represent a portion of a module, segment, or code that includes one or more executable instructions for executing the specified logical functions, and in some alternative embodiments, It should be noted that functions may occur out of order. For example, two successive blocks or steps may actually be performed substantially concurrently, and it is also possible that the blocks or steps are performed in the reverse order of the function as needed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the following embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to enable those skilled in the art to more fully understand the present invention.

1 is a configuration diagram of a file indexing apparatus according to an embodiment of the present invention.

1, the file indexing apparatus 1 includes a folder management section 10, an index list generating section 12, an index information extracting section 14, an index processing section 16, and an index DB 18.

The file indexing device 1 indexes and manages files recorded on a recording medium, for example, a disk. The file indexing device 1 can be implemented as a program for performing indexing and management by executing predetermined processing on a computer. Or a recording medium for recording a program that performs indexing and management.

The folder management unit 10 manages folders to be indexed on a recording medium, for example, a disc. The disk has various folders to be indexed, the folders have a hierarchical structure, and each folder is made up of files. The folder management unit 10 includes a folder list, which is a list of folders to be indexed, for example, the folder lists 1, 2, , n, and the folder list can be set by the administrator. A file includes at least one of text, image, audio, and moving picture data.

The index list generation unit 12 generates a folder to be indexed with respect to the folders managed by the folder management unit 10 and an index list of files in all folders under the index. Each index list can include file creation / modification time and file path.

The index list can be stored temporarily in the file system. The reason is that when the index is re-indexed after the completion of the index, it is used to compare the existing index list with the newly created index list to determine whether or not the index is needed. File paths can be stored in alphabetical order. The reason for storing the file paths in alphabetical order is to facilitate sequential progression when comparing an existing index list with a new index list.

The index information extracting unit 14 extracts the index information from the file using the index list generated through the index list generating unit 12. [ The index information is information on a file to be indexed, the file attribute can be included, and the file content can be further included.

The index information may include common index information common to files and unique index information unique to each file. The common index information is common information regardless of the characteristics of the file, and the unique index information includes different information depending on the characteristics of the file. For example, the common index information may be a file type, a file name, a keyword of the file, a detailed description of the file, a file title, and user information that generated the file. Examples of the unique index information may include at least one of an artist name, an album name, a genre, a reproduction time, a reproduction frequency, an album production year, a lyrics, a copyright, and a format in the case of an audio file.

The characteristics of a file can be obtained by checking the extension of the file path in the index list and the header of the file. In this case, the index information extracting unit 14 can analyze the file characteristics of the file to be indexed, determine which index information to extract, and extract the index information according to the file characteristics. The extension of the file path includes, for example, a video-related extension (AVI, MP4, MPEG, MKV), image extension (BMP, JPG, PNG).

If you need to get a list of newly created indexes and you need to index them, you can insert or update them. At this time, the characteristics of the file to be indexed are grasped and the index information suitable for the identified characteristics is obtained. The characteristics of the file to be indexed can be identified by checking the extension of the file path in the index list and the header of the file. As the index information, the file content corresponds to the body of the document. In this case, only a part of the document content can be fetched as the index information. Body content is also indexed.

The index processing unit 16 uses the generated index list to determine whether or not an index is a necessary object, and generates an index using the index information of the file requiring indexing. The index process of the index processor 16 proceeds as follows based on the above-described technique.

The index processing unit 16 fetches the index lists from the folder list to be indexed. If there are no more index lists to fetch, exit the index.

For folders to be indexed, proceed as follows. If an index list is generated and only the index list generated in the folder list exists, the created list of all the indexes is inserted.

If both the generated index list and the existing index list exist, proceed as follows. In other words, it takes a list of indexes created and takes a list of existing indexes and compares the file paths of two index lists. The result is returned as a number as shown below.

File path of the generated index list> File path of the existing index list: Return positive number

File path of the generated index list <File path of the existing index list: Returns a negative number

File path of generated index list = Return file path of existing index list: 0

The above result value determines whether the index is needed as follows.

If the result of comparing the two file paths is a positive value, the existing index list is deleted, another existing index list is obtained, and the above-described process is performed again.

If the result of comparing the two file paths is negative, insert the generated index list, fetch another generated index list, and perform the above-described process again.

If the result of comparing the two file paths is 0, compare the created index list with the last modification date of the existing index list, and update them if they are different.

If all the generated index lists have been processed, the remaining existing index lists are deleted. If all the existing index lists are processed, the remaining index lists are inserted. The above-described operation is repeated for all the index lists in the folder list to be indexed.

Among the above processes, the following factors may cause the speed decrease.

1. As the number of folders to be indexed increases, the time required to create the index list becomes longer.

2. It takes a long time to fetch the index information of each file to be indexed. For text files, it is sufficient to simply read the data, but most file formats do not. Since most files have their own format, it is necessary to convert them into a form that can be indexed.

3. When indexed contents are reflected in the index database, it takes a long time to approach by simply executing a query in each case. In addition to the actual indexing process, index DB affects performance because it involves locking and unlocking tables, and DB log also being created for each query.

The present invention proposes the following method as a method for eliminating the factors that lower the index speed as described above.

First, when the index list generation unit 12 generates an index list, a thread pool 120 is used. At this time, a thread is created for each folder to generate an index list. As the number of threads increases, the system becomes unstable. Therefore, if the number of threads exceeds the preset number, it can be accessed by entering the standby state. And manages it through the thread pool 120.

When the index list generation unit 12 stores the generated index list, the index list generation unit 12 stores the generated index list for each folder in charge of each thread, not in one file. If an asynchronous method is used to store an index list in a single file, a race condition problem occurs and the storage order can not always be guaranteed. Since at least the order of archiving is fixed by folder, do not keep it as one file, but keep the file by folder.

Second, the index information extraction unit 14 and the index processing unit 16 perform operations in an asynchronous manner. Index performance will be improved if the index processor 16 can simultaneously perform processing on other files to be indexed while the index information extractor 14 obtains the index information of the files to be indexed. There is no association between the two tasks, so access is possible in a sufficiently asynchronous manner.

According to the asynchronous method according to the embodiment, the index information extracting unit 14 performs an operation through a thread pool 140 composed of a plurality of producer threads, and the index processing unit 16 includes a single consumer thread 162, . &Lt; / RTI &gt; Since the time to obtain the index information of the file is not uniform, this task is performed through separate producer threads. When the index information generation job is completed, each producer thread stores index information in a waiting queue 160 provided by the index processing unit 16. [ The index processor 16 has its own wait queue 160 and monitors whether the consumer thread 162 periodically has content in the wait queue 160. [ If the index information is included in the waiting queue 160, the index information is fetched and index processing is performed. The index information extracting unit 14 manages the producer thread pool 140 because the system thread may become unstable if too many producer threads are created. That is, when the number of producer threads constituting the producer thread pool 140 exceeds a predetermined number, the processor enters the standby state.

Third, the index processing unit 16 uses an index query having a batch index function. The index processing unit 16 creates a temporary folder and adds the contents to be indexed into the temporary folder. At this time, the number of additional records is increased by one. The number of additional records should be managed in memory. If the number of added records exceeds a preset number, the index processing unit 16 calls an index query to perform a batch index. If the query is successful, delete the temporary folder and initialize the number of records added.

Fourth, the index processing unit 16 divides the handle for accessing the index DB 18 into read-only and write-only. Batch index is dramatically faster than processing by calling an index query per query, but it is not processed immediately because it processes multiple records at once. Therefore, if you set both handles to read and write on one handle, the query invoked for search will wait until the batch index is finished. So, separate the accessible handles of the table in the index DB (18) as read-only and write-only, and use the appropriate handle according to the purpose of the query.

2 is a flowchart illustrating a file indexing method according to an embodiment of the present invention.

Referring to FIGS. 1 and 2, the file indexing device 1 creates a folder list in folders to be indexed in the disc (210). Subsequently, the file indexing device 1 generates an index list of files in the folder (220). The index list can include the file's last modified date and file path information. In generating an index list, the file indexing device 1 can generate an index list on a folder basis through a plurality of threads constituting the thread pool, and store the generated index list in a folder unit. At this time, the number of threads constituting the thread pool is managed, and when the number of threads exceeds a preset number, it is possible to enter the waiting state.

Subsequently, the file indexing device 1 extracts the index information of the file from the generated index list (230). The file indexing device 1 analyzes file characteristics of a file to be indexed, determines which index information to extract, and extracts index information matching the file characteristics. The file indexing device 1 according to an embodiment extracts index information through a producer thread pool including a plurality of producer threads, and stores the extracted index information in a waiting queue managed by a consumer thread. At this time, the number of producer threads constituting the producer thread pool is managed, and when the number of threads exceeds the preset number, the processor can enter the waiting state.

Subsequently, the file indexing device 1 generates an index using the index information of the file requiring indexing (240). At this time, while the file indexing device 1 extracts the index information by the producer thread, the consumer thread simultaneously performs the indexing on the other file in which the index information extraction is performed. The file indexing device 1 according to an embodiment monitors whether or not index information is present in a waiting queue and, if index information exists, obtains index information and performs an index process to generate an index. The generated index is stored in the index DB.

The file indexing device 1 according to the embodiment creates a temporary folder, adds contents to be indexed in the temporary folder to the record, and increases the number of additional records by one. At this time, if the number of additional records exceeds the preset number, the index query is called to perform the indexing. If the query is successful, the temporary folder is deleted and the number of additional records is initialized.

The file indexing device 1 according to the embodiment separates the handles which can be accessed in the table in the index DB into the handles for reading and the handles for writing, and indexes them by using the handles corresponding to the purpose of the query.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

1: file indexing device 10: folder managing part
12: Index list generation unit 14: Index information extraction unit
16: Index processor 18: Index DB

Claims (10)

An index list generating unit for generating an index list of files in a folder to be indexed;
An index information extracting unit for extracting index information of the file from the generated index list; And
The index information extracting unit extracts the index information of the file that is to be indexed by using the generated index list and determines whether or not the index file is an object to be indexed. An index processing unit for simultaneously performing indexing on the indexes;
The file indexing device comprising: The method according to claim 1,
Wherein the index list includes a file last modification date and file path information. The apparatus of claim 1, wherein the index list generation unit
A thread pool including a plurality of threads for generating an index list on a folder basis and storing the generated index list on a folder basis;
The file indexing device comprising: 4. The apparatus of claim 3, wherein the index list generation unit
And manages the number of threads constituting the thread pool, and enters the waiting state when the number of threads exceeds a predetermined number. The apparatus of claim 1, wherein the index information extracting unit
Wherein the file indexing device analyzes the file characteristics of the file to be indexed to determine which index information is to be extracted and extracts index information corresponding to the file property. The apparatus of claim 1, wherein the index information extracting unit
A producer thread pool including a plurality of producer threads for extracting index information and providing the extracted index information to the index processor;
The file indexing device comprising: 7. The apparatus of claim 6, wherein the index information extracting unit
And manages the number of producer threads constituting the producer thread pool and enters the waiting state when the number of threads exceeds a predetermined number. The image processing apparatus according to claim 1, wherein the index processing unit
A waiting queue storing at least one index information extracted by the index information extracting unit; And
A consumer thread monitoring whether or not index information exists in the waiting queue, fetching index information if index information exists, generating an index by performing index processing, and storing the generated index in an index DB;
The file indexing device comprising: The image processing apparatus according to claim 1, wherein the index processing unit
If the number of additional records exceeds the preset number, the index query is called to perform batch indexing. If the query execution is successful, the temporary folder is deleted and the number of additional records is incremented And initializes the file indexing device. The image processing apparatus according to claim 1, wherein the index processing unit
A file indexing device for separating a handle accessible to a table in the index DB into a read handle and a write handle and using the handle corresponding to the purpose of the query.

Images