Summary of the invention
At above problem, the invention provides a kind of information merging method and system, from a plurality of original data storage unit, obtain the problem that information efficiency is low and the information releasing existence is too much redundant in the prior art in order to solve.
For addressing the above problem, a kind of information merging method provided by the invention is applied to the system that comprises centre data storage unit, user terminal and be used for storing the original data storage unit of raw data more than one, said method comprising the steps of:
A, user terminal obtain raw data from described original data storage unit;
B, user terminal carry out merger with the described raw data of obtaining;
C, the user terminal data storage after with merger is to the centre data storage unit.
User terminal obtains raw data by multithreading in the steps A from described original data storage unit.
User terminal adopts batch to read mode and obtain raw data from described original data storage unit in the steps A.
Described system further comprises merger data storage cell, data buffer unit and temporary data storage unit, and then the described user terminal of step B step that the described raw data of obtaining is carried out merger is:
B1, user terminal are set up a merger tables of data for each original data storage unit in the merger data storage cell, the field of this merger tables of data comprises all fields and merger tag field in the corresponding original data storage unit, the original data storage that to obtain from the original data storage unit and is initialized as not merger with the merger tag field content of this merger tables of data to the merger tables of data of correspondence;
The merger tag field content modification of B2, data that user terminal is identical with major key content in the temporary data storage unit with major key content in the described merger tables of data is merger;
B3, user terminal to data buffer unit, carry out merger to data with the data storage in the described merger tables of data in data buffer unit.
User terminal is before a merger tables of data is set up in each original data storage unit, to remove the data in merger data storage cell and the data buffer unit in the merger data storage cell among the step B1.
Step B3 comprises:
B31, user terminal are set up tables of data in data buffer unit, the field of this tables of data comprises all fields in the merger tables of data;
B32, user terminal arrive described tables of data according to predefined merger rule with the data storage in the described merger tables of data;
B33, user terminal with the field in the predefined original data storage unit as the merger field, is data according to previously selected operation rule and this merger field with all aggregation of data that satisfy operation rule in the described tables of data, described all data that satisfy operation rule as the data that are merged, are deleted the described data that are merged in data buffer unit.
The described merger rule of step B32 comprises increment merger or brand-new merger.
The described operation rule of step B33 comprises equal merger, comprises merger or equals merger.
Before in data buffer unit, deleting the described data that are merged among the step B33, the merger tag field content of the described data that are merged is set to merger in the merger data storage cell, and is that the data storage of merger is to temporary data storage unit with merger tag field content in the merger data storage cell.
A kind of information merge system, this system comprises:
More than one original data storage unit, described original data storage unit is used to store raw data;
User terminal is used for obtaining raw data from described original data storage unit, and the raw data of obtaining is carried out merger, and the data storage after the merger is arrived the centre data storage unit;
The centre data storage unit is used to store the data after the described merger.
Described system further comprises:
The merger data storage cell is used to store the described raw data of obtaining from the original data storage unit;
Temporary data storage unit is used for storing described merger data storage cell merger tag field content and is the data of merger;
Data buffer unit, the data that are used for storing described merger data storage cell.
Described user terminal comprises:
The thread reading unit is used for obtaining raw data from described original data storage unit;
The merger processing unit, the raw data that is used for obtaining is carried out merger;
Storage element is used for the data storage after the described merger to the centre data storage unit as a result.
Described merger processing unit comprises:
Storage unit, be used for removing the data of merger data storage cell and data buffer unit, in the merger data storage cell, set up a merger tables of data for each original data storage unit, the field of this merger tables of data comprises all fields and merger tag field in the corresponding original data storage unit, the original data storage that to obtain from the original data storage unit and is initialized as not merger with the merger tag field content of this merger tables of data to the merger tables of data of correspondence;
Revise the unit, the merger tag field content modification that is used for the data that described merger tables of data major key content is identical with major key content in the temporary data storage unit is merger;
The merger unit is used for data storage with described merger tables of data to data buffer unit, in data buffer unit data is carried out merger.
Described merger unit comprises:
Preserve the unit, be used for setting up tables of data in data buffer unit, the field of this tables of data comprises all fields in the merger tables of data;
Reading unit is used for according to predefined merger rule the data storage of described merger tables of data to described tables of data;
Processing unit, be used for field with predefined original data storage unit as the merger field, is data according to previously selected operation rule and this merger field with all aggregation of data that satisfy operation rule in the described tables of data, satisfy all data of operation rule as the data that are merged with described, the merger tag field content of the described data that are merged is set to merger in the merger data storage cell, and with merger tag field content in the merger data storage cell be the data storage of merger to temporary data storage unit, the described data that are merged of deletion in data buffer unit.
Among the present invention, user terminal obtains data and the data of obtaining is carried out storing the centre data storage unit into after the merger from a plurality of original data storage unit, directly from this centre data storage unit, obtain in the time of need obtaining data, adopt method of the present invention, avoided all from a plurality of original data storage unit, obtaining data at every turn, improve efficient, simultaneously similar data have been merged, improved the integrality of data.
Embodiment
A kind of information merging method provided by the invention, be applied to comprise the system of the original data storage unit, centre data storage unit, merger data storage cell, temporary data storage unit, data buffer unit and the user terminal that are used for storing raw data more than, the core concept of this method is to obtain the raw data in a plurality of original data storage unit and the data of obtaining are carried out merger, deposit the data after the merger in the centre data storage unit, directly visit this centre data storage unit in the time of need obtaining data.
Referring to Fig. 1, the step of a kind of information merging method provided by the invention comprises:
101, user terminal obtains raw data in the mode that reads in batches by multithreading from the original data storage unit;
102, user terminal carries out merger with the described raw data of obtaining;
In this step, at first, user terminal is removed the data in merger data storage cell and the data buffer unit, in the merger data storage cell, set up a merger tables of data for each original data storage unit, the field of this merger tables of data comprises all fields and the merger tag field in the corresponding original data storage unit, the original data storage that to obtain from the original data storage unit and is initialized as not merger with the merger tag field content of this merger tables of data in the merger tables of data of correspondence;
Then, the merger tag field content modification of the data that user terminal is identical with major key content in the temporary data storage unit with major key content in the described merger tables of data is merger;
Then, user terminal is set up tables of data in data buffer unit, the field of this tables of data comprises all fields in the merger tables of data, according to predefined merger rule the data storage in the described merger tables of data is arrived described tables of data again, described merger rule comprises increment merger or brand-new merger, if predefined merger rule is the increment merger, the data storage that then with merger tag field content in the described merger tables of data is not merger is to described tables of data, if predefined merger rule is brand-new merger, then all data storage in the described merger tables of data are arrived described tables of data;
Next, user terminal is data with predefined operation rule and merger field with all aggregation of data that satisfy operation rule in the described tables of data, and other field contents beyond the merger field superpose as the content of field after the merger; Satisfy all data of operation rule as the data that are merged with described, the merger tag field content of the described data that are merged is set to merger in the merger data storage cell, and with merger tag field content in the merger data storage cell be the data storage of merger to temporary data storage unit, described data that are merged of deletion in data buffer unit at last;
Described predefined operation rule comprises equal merger, comprises merger or equals merger, equating wherein that merger is meant carries out merger when having the identical data of two above merger field contents in the described data buffer unit, the merger step is: with described two identical aggregation of data of above merger field contents is data, and other field contents beyond the merger field superpose as the content of field after the merger; Comprise and carry out merger when merger is meant the data that exist two above merger field contents all to comprise a certain certain content in the described data buffer unit, this certain content is imported by the user, the merger step is: the aggregation of data that described two above merger field contents are all comprised a certain certain content is data, and other field contents beyond the merger field superpose as the content of field after the merger; Equaling merger is meant in the described data buffer unit and carries out merger when existing two above merger field contents to be equal to the data of a certain certain content, this certain content is imported by the user, the merger step is: the aggregation of data that described two above merger field contents are equal to a certain certain content is data, and other field contents beyond the merger field superpose as the content of field after the merger.Described merger field is any field in the predefined original data storage unit;
103, the data storage after the merger in the above-mentioned data buffer unit is arrived the centre data storage unit.
When needs obtain data from more than one original data storage unit, directly visit the centre data storage unit, from this centre data storage unit, obtain data.
Below in conjunction with specific embodiment a kind of information merging method provided by the invention is described.
The customer information of supposing 5 departments an of enterprise leaves original data storage unit 1 respectively in, original data storage unit 2, original data storage unit 3, in original data storage unit 4 and the original data storage unit 5, every customer information includes user ID, name, sex, age, phone and address field, wherein major key is a user identification field, predefined merger field is the address field in the original data storage unit, predefined merger rule is the increment merger, operation rule is for comprising merger, and the merger process to the customer information of these 5 original data storage unit is as follows so:
1, user terminal obtains raw data in the mode that reads in batches by multithreading from each original data storage unit;
2, at first, user terminal is removed the data in merger data storage cell and the data buffer unit, for setting up a merger tables of data in each original data storage unit, the field of this merger tables of data comprises user ID, name, sex, age, phone, address field and merger tag field, the described original data storage of obtaining from the original data storage unit in the merger tables of data of correspondence, and is initialized as not merger with the merger tag field content of this merger tables of data; Then, the merger tag field content modification of the data that user terminal is identical with user identification field content in the temporary data storage unit with user identification field content in the described merger tables of data is merger, setting up the tables of data comprise user ID, name, sex, age, phone, address and merger tag field in data buffer unit, is that the data storage of not merger is in this tables of data according to previously selected merger rule with merger tag field content in the described merger tables of data; Then, user terminal carries out merger according to previously selected operation rule and merger field with the data in the described tables of data, even exist two above address field contents to comprise the data of a certain certain content in the tables of data, this certain content is imported by the user, the certain content of supposing user's input is " upward ", the aggregation of data that then all address field contents in the tables of data is comprised " upward " is data, and other field contents beyond the address field are superposeed as the content of field after the merger; At last, the merger tag field content that address field content described in the merger data storage cell comprises the data of " upward " is set to merger, with merger tag field content in the merger data storage cell be the data storage of merger to temporary data storage unit, again these data are deleted in data buffer unit;
3, user terminal arrives the centre data storage unit with the data storage in the described data buffer unit.
When needs obtain data from a plurality of original data storage unit, directly visit the centre data storage unit, from this centre data storage unit, obtain data.
Referring to Fig. 2, the present invention also provides a kind of information merge system, and this system comprises:
More than one original data storage unit 21, described original data storage unit is used to store raw data;
User terminal 22 is used for obtaining raw data from described original data storage unit, and the raw data of obtaining is carried out merger, and the data storage after the merger is arrived the centre data storage unit;
Centre data storage unit 23 is used to store the data after the described merger;
Merger data storage cell 24 is used to store the described raw data of obtaining from the original data storage unit;
Data buffer unit 25, the data that are used for storing described merger data storage cell;
Temporary data storage unit 26 is used for storing described merger data storage cell merger tag field content and is the data of merger.
Described user terminal comprises:
Thread reading unit 221 is used for obtaining raw data from described original data storage unit;
Merger processing unit 222, the raw data that is used for obtaining is carried out merger;
Storage element 223 as a result, are used for the data storage after the described merger to the centre data storage unit.
Described merger processing unit comprises:
Storage unit 2221, be used for removing the data of merger data storage cell and data buffer unit, in the merger data storage cell, set up a merger tables of data for each original data storage unit, the field of this merger tables of data comprises all fields and merger tag field in the corresponding original data storage unit, the original data storage that to obtain from the original data storage unit and is initialized as not merger with the merger tag field content of this merger tables of data to the merger tables of data of correspondence;
Revise unit 2222, the merger tag field content modification that is used for the data that described merger tables of data major key content is identical with major key content in the temporary data storage unit is merger;
Merger unit 2223 is used for data storage with described merger tables of data to data buffer unit, in data buffer unit data is carried out merger.
Described merger unit comprises:
Preserve unit 22231, be used for setting up tables of data in data buffer unit, the field of this tables of data comprises all fields in the merger tables of data;
Reading unit 22232 is used for according to predefined merger rule the data storage of described merger tables of data to described tables of data;
Processing unit 22233, be used for field with the original data storage unit as the merger field, is data according to previously selected operation rule and this merger field with all aggregation of data that satisfy operation rule in the described tables of data, satisfy all data of operation rule as the data that are merged with described, the merger tag field content of the described data that are merged is set to merger in the merger data storage cell, and with merger tag field content in the merger data storage cell be the data storage of merger to temporary data storage unit, the described data that are merged of deletion in data buffer unit.
Obviously, those skilled in the art can carry out various changes and modification to the present invention and not break away from the spirit and scope of the present invention.Like this, if of the present invention these are revised and modification belongs within the scope of claim of the present invention and equivalent technologies thereof, then the present invention also is intended to comprise these changes and modification interior.