CN107679141A - Data storage method, device, equipment and computer-readable recording medium - Google Patents
Data storage method, device, equipment and computer-readable recording medium Download PDFInfo
- Publication number
- CN107679141A CN107679141A CN201710873319.9A CN201710873319A CN107679141A CN 107679141 A CN107679141 A CN 107679141A CN 201710873319 A CN201710873319 A CN 201710873319A CN 107679141 A CN107679141 A CN 107679141A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- call
- initial
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data storage method, this method includes：Obtain has the input sample data of same data structure and corresponding database template data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data with data source；The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；The initial data of the data source is converted to by corresponding data storage by the Data Translation model, and the data storage is imported into the database.The invention also discloses a kind of data loading device, equipment and computer-readable recording medium.The present invention carries out analysis contrast by the data structure to data source and the data structure of system database, build Data Translation model, and data loading is automatically performed by the Data Translation model, so as to accomplish that the automatic storage of data can be achieved in no manual intervention, improve operating efficiency.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of data storage method, device, equipment and computer
Readable storage medium storing program for executing.
Background technology
With the development of technology, start the methods of risk assessment based on computer air control engine, this side occur on the market
Method is to accumulate substantial amounts of user data by internet, and draws assessment result according to these data and computer air control model,
And then decide whether to make loans.
, it is necessary to which a large amount of initial data for first obtaining these on network import when using this big data appraisal procedure
Into system database.But because the sources of these data are various, it has, and structure is different, data redundancy degree is high, non-
The features such as linear；And the data memory format required by database is often different with these raw data formats, this is resulted in
Data loading process it is extremely cumbersome, it is necessary to staff the form of legacy data is manually converted into system data place will
The form asked, increase staff by the workload of data loading.
The content of the invention
It is a primary object of the present invention to provide a kind of data storage method, device, equipment and computer-readable storage medium
Matter, it is intended to which solution realizes that data are put in storage automatically, improves operating efficiency.
To achieve the above object, the present invention provides a kind of data storage method, and the data storage method includes following step
Suddenly：
Obtain has the input sample data of same data structure and corresponding database template data, and root with data source
According to Data Translation model corresponding to the input sample data and the generation of database template data；
The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；
The initial data of the data source is converted to by corresponding data storage by the Data Translation model, and by institute
State data storage and import the database.
Alternatively, the Data Translation model according to corresponding to the input sample data and the generation of database template data
The step of include：
The input sample data and database template data as input training set and are exported into training set respectively, and base
Analyzing and training is carried out to initial machine learning model in the input training set and output training set, construction obtains data conversion mould
Type, wherein the Data Translation model includes conversion logic between the input sample data and template data.
Alternatively, the data storage includes data type mark,
It is described that the initial data of the data source is converted to by corresponding data storage by the Data Translation model, and
After the step of data storage is imported into the database, in addition to：
Similar data storage is searched according to data type mark in the database, and by the similar storage number
According to geometric operation is carried out, assemblage characteristic data are obtained, to be extended to data dimension.
Alternatively, it is described that the initial data of the data source is converted to by corresponding storage by the Data Translation model
Data, and by the data storage import the database the step of after, in addition to：
The data directory file of the database is updated according to the importing situation of the data storage.
Alternatively, it is described that the initial data of the data source is converted to by corresponding storage by the Data Translation model
Data, and by the data storage import the database the step of after, in addition to：
When receiving data call request, the data call interface of the database is distributed to data call end, for
The data call end carries out data call.
Alternatively, described when receiving data call request, the data that the database is distributed to data call end are adjusted
With interface, the step of carrying out data call for the data call end, includes：
When receiving data call request, the data call interface of the database is distributed to data call end, and lead to
Cross the data call interface and obtain the script file that the data call end is sent；
The call operation according to corresponding to performing the script file, to obtain calling data in the database, and will
The calling data are sent to the data.
Alternatively, described when receiving data call request, the data that the database is distributed to data call end are adjusted
With interface, the step of carrying out data call for the data call end, includes：
When receiving data call request, the identity in being asked according to the data call is to the data call
The identity at end is verified；
When the identity at the data call end passes through checking, the data call of the database is distributed to data call end
Interface, so that the data call end carries out data call.
In addition, to achieve the above object, the present invention also provides a kind of data loading device, the data loading device bag
Include：
Model generation module, there are the input sample data of same data structure and corresponding number with data source for obtaining
According to library template data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；
Data input module, inputted for obtaining the initial data of the data source, and by the initial data to described
Data Translation model；
Data import modul, for being converted to the initial data of the data source correspondingly by the Data Translation model
Data storage, and the data storage is imported into the database.
In addition, to achieve the above object, the present invention also provides a kind of data loading equipment, and the data loading equipment includes
Processor, memory and it is stored on the memory and can be by the data loading program of the computing device, wherein institute
When stating data loading program by the computing device, the step of realizing data storage method described above.
In addition, to achieve the above object, the present invention also provides a kind of computer-readable recording medium, described computer-readable
Data loading program is stored with storage medium, wherein when the data loading program is executed by processor, is realized described above
The step of data storage method.
The present invention has the input sample data of same data structure and corresponding database mould by obtaining with data source
Plate data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；Described in acquisition
The initial data of data source, and the initial data is inputted to the Data Translation model；Pass through the Data Translation model
The initial data of the data source is converted into corresponding data storage, and the data storage is imported into the database.It is logical
Cross and analysis contrast, structure are carried out by the data structure to data source and the data structure of system database with upper type, the present invention
Data Translation model is built, and by the Data Translation model scan data source interface, to data structure progress autonomous classification, and certainly
It is dynamic to complete data loading, from without being artificially identified, parsing to initial data and further encapsulate, and pass through computer
Model is transferred automatically, is accomplished that the automatic storage of data can be achieved in no manual intervention, is reduced the artificial behaviour of data loading
Measure, improve operating efficiency.
Brief description of the drawings
Fig. 1 is the hardware architecture diagram for the data loading equipment that scheme of the embodiment of the present invention is related to；
Fig. 2 is the schematic flow sheet of data storage method first embodiment of the present invention；
Fig. 3 is the schematic flow sheet of data storage method second embodiment of the present invention；
Fig. 4 is the schematic flow sheet of data storage method 3rd embodiment of the present invention；
Fig. 5 is the schematic flow sheet of data storage method fourth embodiment of the present invention；
Fig. 6 is the high-level schematic functional block diagram of data loading device first embodiment of the present invention.
The realization, functional characteristics and advantage of the object of the invention will be described further referring to the drawings in conjunction with the embodiments.
Embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
The main thought of scheme of the embodiment of the present invention is：Obtain the input sample number that there is same data structure with data source
According to corresponding database template data, and according to the input sample data and database template data generation corresponding to data
Transformation model；The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；Pass through
The initial data of the data source is converted to corresponding data storage by the Data Translation model, and the data storage is led
Enter the database.
The present embodiments relate to data storage method be mainly used in data loading equipment.
Reference picture 1, Fig. 1 are the hardware architecture diagram for the data loading equipment being related in scheme of the embodiment of the present invention.This
In inventive embodiments, data loading equipment can include processor 1001 (such as CPU), communication bus 1002, user interface
1003, network interface 1004, memory 1005.Wherein, communication bus 1002 is used to realize the connection communication between these components；
User interface 1003 can include display screen (Display), input block such as keyboard (Keyboard)；Network interface 1004 can
Wireline interface, the wave point (such as WI-FI interfaces) that can include standard of choosing；Memory 1005 can be high-speed RAM storage
Device or stable memory (non-volatile memory), such as magnetic disk storage, memory 1005 are optional
It can also be the storage device independently of aforementioned processor 1001.
Alternatively, the data loading equipment can also include camera, RF (Radio Frequency, radio frequency) circuit, pass
Sensor, voicefrequency circuit, WiFi module etc..Wherein, sensor ratio such as optical sensor, motion sensor and other sensors.
Specifically, optical sensor may include ambient light sensor, and ambient light sensor can adjust display according to the light and shade of ambient light
The brightness of screen.As one kind of motion sensor, gravity accelerometer can detect in all directions (generally three axles) and add
The size of speed, size and the direction of gravity are can detect that when static, the application (ratio of library facilities posture is entered available for identification data
Such as horizontal/vertical screen switching, magnetometer pose calibrating), Vibration identification correlation function (such as pedometer, tap)；Certainly, the data
Enter library facilities and can also configure the other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared ray sensor, herein no longer
Repeat.
It will be understood by those skilled in the art that the hardware configuration of the data loading equipment shown in Fig. 1 does not form logarithm
According to the restriction for entering library facilities, it can include than illustrating more or less parts, either combine some parts or different portions
Part is arranged.
With continued reference to Fig. 1, the memory 1005 in Fig. 1 as a kind of computer-readable recording medium can include operation system
System, network communication module and data loading program.
In Fig. 1, network communication module is mainly used in connecting database, enters row data communication with database；And processor
1001 can call the data loading program stored in memory 1005, and perform following operate：
Obtain has the input sample data of same data structure and corresponding database template data, and root with data source
According to Data Translation model corresponding to the input sample data and the generation of database template data；
The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；
The initial data of the data source is converted to by corresponding data storage by the Data Translation model, and by institute
State data storage and import the database.
Further, the data conversion mould according to corresponding to the input sample data and the generation of database template data
The step of type, includes：
The input sample data and database template data as input training set and are exported into training set respectively, and base
Analyzing and training is carried out to initial machine learning model in the input training set and output training set, construction obtains data conversion mould
Type, wherein the Data Translation model includes conversion logic between the input sample data and template data.
Further, the data storage includes data type mark, and processor 1001 may call upon memory
The data loading program stored in 1005, and perform following operate：
Similar data storage is searched according to data type mark in the database, and by the similar storage number
According to geometric operation is carried out, assemblage characteristic data are obtained, to be extended to data dimension.
Further, processor 1001 may call upon the data loading program stored in memory 1005, and perform with
Lower operation：
The data directory file of the database is updated according to the importing situation of the data storage.
Further, processor 1001 may call upon the data loading program stored in memory 1005, and perform with
Lower operation：
When receiving data call request, the data call interface of the database is distributed to data call end, for
The data call end carries out data call.
Further, it is described when receiving data call request, the data of the database are distributed to data call end
Calling interface, so that the step of data call end progress data call includes：
When receiving data call request, the data call interface of the database is distributed to data call end, and lead to
Cross the data call interface and obtain the script file that the data call end is sent；
The call operation according to corresponding to performing the script file, to obtain calling data in the database, and will
The calling data are sent to the data.
Further, it is described when receiving data call request, the data of the database are distributed to data call end
Calling interface, so that the step of data call end progress data call includes：
When receiving data call request, the identity in being asked according to the data call is to the data call
The identity at end is verified；
When the identity at the data call end passes through checking, the data call of the database is distributed to data call end
Interface, so that the data call end carries out data call.
Based on the hardware configuration of above-mentioned data loading equipment, each embodiment of data storage method of the present invention is proposed.
The present invention provides a kind of data storage method.
Reference picture 2, Fig. 2 are the schematic flow sheet of data storage method first embodiment of the present invention.
In the present embodiment, the data storage method comprises the following steps：
Step S10, obtain has the input sample data of same data structure and corresponding database template with data source
Data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；
User's initial data of internet, it is (the example for being stored and being transmitted by the computer language of structuring mostly
Such as xml, json and binary system data format).But because the sources of these initial data are various, different data sources it
Between often take the same data structure, such as the user card punching consumption data obtained from A banks is stored using xml forms
, the user telephone fee billing data obtained from B communication common carriers be collection json forms store；Therefore between these initial data
Have the characteristics that structure is different, data redundancy degree is high, non-linear；And the data memory format required by database often with
These raw data formats are different.In this case, if the initial data of unprocessed form is stored in into system database,
Firstly the need of the form be converted to the form of initial data required by system database, that is to say, that need to initial data
Initial data structure is analyzed, it is determined that under such a structure type, the effective information of initial data is in whole data
Which part of file, it is by which partial content deposit system database of initial data when carrying out data loading.
Specifically, first having to analyze the initial data structure of data source in the present embodiment, initial data knot is determined
Implication in structure per one piece of data.What deserves to be explained is currently used structuring computer language (such as xml, json and two
The data formats such as system), it has succinct and clearly hierarchical structure, and for these data, its main data object is often
Specific mark form (or structure, identifier etc.) can be accompanied by be marked.For example, the data of xml forms, are contained a large amount of
Start-tag and end-tag be used for flag data content；The data of json forms, then it is come table by " { } " and " [] " group
Show data object.Based on the above situation, whole file traversal can be carried out to initial data, according to its specific mark format identification
The effective information gone out in data, and the data storage for meeting database storage requirement is generated according to the effective information.
In the present embodiment, it is contemplated that the data volume of initial data is more huge, and the full text of initial data travels through and effective
The actions such as information identification are again relatively single and cumbersome, therefore can be trained by the way of machine learning and established out phase
The Data Translation model answered, and the importation of Data Translation model is docked with data source, pass through Data Translation model
Initial data is completed to the transfer process of data storage.Wherein, machine learning refers to carry out summing up experience independent of the mankind, inputs and patrol
Volume, the mankind only need mass data (including input data and output data) to input to computer, then by computer oneself
Data switching logic or Mapping and Converting relation therein is summed up, summarizes corresponding logical code, so as to obtain a data
Transformation model.Specifically, in the present embodiment, the input sample data that there is same data structure with data source are first obtained, this
Input sample data can be directly being obtained from data source or artificial according to the data structure of data source progress structure
Make；Also obtaining database template data corresponding to this input sample data simultaneously, (i.e. the data of input sample data turn
Change result)；Then using the input sample data as training set is inputted, using database template data as output training set, and
The input training set and output training set are inputted to original computer learning model, trained based on the input training set and output
Set pair initial machine learning model carries out analyzing and training, by computer automatically to both data structure forms and data content etc.
Aspect is contrasted and analyzed, and sums up conversion rule, summarizes conversion logic, draws transformation model (including conversion logic).
Such as input sample data are A, database template data are B, and A and B are inputted to initial machine learning model, by computer certainly
Oneself goes to determine conversion logic X according to input and output, to cause A->X->B.
The Data Translation model and conversion logic rule, staff obtained certainly for training can be manually adjusted
Or modification；Initial model can be also modified according to actual use situation, it is perfect to be carried out to model.
Step S20, the initial data of the data source is obtained, and the initial data is inputted to the data conversion mould
Type；
After Data Translation model is obtained, you can carry out data conversion and data loading using the Data Translation model.
Now, the input of Data Translation model can be docked with data source by the data-interface that data source provides, so as to obtain
The initial data of data source, and the initial data is inputted into Data Translation model and carries out data conversion.
Step S30, the initial data of the data source is converted to by corresponding storage number by the Data Translation model
According to, and the data storage is imported into the database.
Data Translation model is when getting the initial data of data source, you can the conversion logic obtained according to training study
Initial data is changed, so as to obtain the data storage for meeting database storage requirement.
In the present embodiment, Data Translation model can be realized with scala language.Data loading model first will be right
The interface of data source is parsed, and obtains the address of data source, and is docked according to the address with data source, such as：
def ScanApi(val api):Int{}
Wherein, val api are data resource interface address, and the address is a variable, this variable, it may be possible to one
Url addresses, it is also possible to the path values of other forms.When being docked to data source according to the address, you can get the data
The initial data in source.By taking the initial data of json forms as an example, now these will be got from data source by http agreements
Initial data, then have
Val json=http.get (api)
When getting the initial data data, the logical laws according to summary are known to the form of initial data
Not, judge which type the form of initial data belongs to；It is determined that initial data data format when, this type will be selected
Corresponding resolution rules are parsed to initial data, and effective information (the demand letter of database is extracted from initial data
Breath), so as to obtain data object val obj, i.e.,
Val obj=createObj (json)
What deserves to be explained is if nested subdata in initial data, subnumber that can be by recursive function nesting
According to the bottom is resolved to, ensure the valid data integrality of initial data transfer process.
When obtaining effective information, you can by translate functions, corresponding sql sentences are write, so as to obtain
Val sql sentence set, such as
Val sql=translate (obj)
Related data write operation is now carried out by val sql sentences again, that is, realizes the automatic storage of data, such as
Val db=touch (sql)
return db.write(sql)
What deserves to be explained is for completing initial data to the conversion process of data storage, also include some contents certainly
Delete, the format code as necessary to the syntax rule for leaving out some original data sources, or leave out some data places not
The content information wanted.
Certainly, in specific implementation, data loading model and data conversion code can be realized by other language, or
It is the other forms realization by scala language；Said process is only the data conversion function generation of data loading model and correlation
Code is not for example, represent the restriction to the present embodiment technical scheme.
Obtained in the present embodiment has the input sample data of same data structure and corresponding database mould with data source
Plate data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；Described in acquisition
The initial data of data source, and the initial data is inputted to the Data Translation model；Pass through the Data Translation model
The initial data of the data source is converted into corresponding data storage, and the data storage is imported into the database.It is logical
Cross and analysis contrast carried out by the data structure to data source and the data structure of system database with upper type, the present embodiment,
Data Translation model is built, and by the Data Translation model scan data source interface, autonomous classification is carried out to data structure, and
Data loading is automatically performed, from without being artificially identified, parsing to initial data and further encapsulate, and passes through calculating
Machine model is transferred automatically, is accomplished that the automatic storage of data can be achieved in no manual intervention, is reduced the artificial of data loading
Operational ton, improve operating efficiency.
Reference picture 3, Fig. 3 are the schematic flow sheet of data storage method second embodiment of the present invention.
Based on above-mentioned embodiment illustrated in fig. 2, the data storage in the present embodiment identifies including data type, the step
After S30, in addition to：
Step S40, similar data storage is searched according to data type mark in the database, and will be described same
Class data storage carries out geometric operation, assemblage characteristic data is obtained, to be extended to data dimension.
, will also be when being changed by Data Translation model to initial data, obtaining data storage in the present embodiment
Data type mark is added in data storage, wherein data type mark can be used for data type is marked.According to the number
According to type identification, the similar data storage in database can be found out, and these similar data storages are subjected to geometric operation,
On the basis of original data storage, assemblage characteristic data are obtained, in other words, to from different data sources, description same target
Different aspect data, geometric operation can be carried out to its data value, the integrated data value for describing the object be obtained, so as to right
Data dimension is extended.
For example, when data source obtains initial data, obtain user human from bank and pay number in the charges for water and electricity of September part
Telephone expenses expenditure data bs of the user human in September part is obtained according to a, and from telecommunications company；By Data Translation model to data a and
Data b is changed, obtain corresponding to data storage A and B, while data type mark is further comprises in A and B, using mark A as
User human is that user human pays data in the telephone expenses of September part in the charges for water and electricity data of September part, B；Now, due to A and B
Identified with similar data type, all have recorded branch artificial situations of the user human in September part different in terms of, therefore, can
A and B are combined, obtain assemblage characteristic data C, it is monthly in the water power of September part, telephone expenses that the C represents user user human
Total expenditure (i.e. C=A+B).Further, to obtain user's human7 months, August, the trimestral water power of September, telephone expenses monthly total
When paying data, three data can also be carried out with geometric operation again, obtain user human water power, the total expenditure of telephone expenses season
Data, so as in a manner of tree-shaped combination, be extended to data dimension.
, can after the data storage of unified form is converted raw data into by Data Translation model in the present embodiment
Geometric operation is carried out to the data storage of same type or same target, multiple data are integrated, obtains data splitting, from
And data dimension is extended, the data content in database is enriched, facilitates related personnel to be got more from database
Multi information.
Reference picture 4, Fig. 4 are the schematic flow sheet of data storage method 3rd embodiment of the present invention.
Based on above-mentioned Fig. 2 or embodiment illustrated in fig. 3, in the present embodiment, after step S30, in addition to：
Step S50, the data directory file of the database is updated according to the importing situation of the data storage.
In the present embodiment, when obtaining data storage and the data storage being imported into data, for convenience to data storage
Importing situation recorded, can also generate corresponding data indexing information, and according to the information to the data rope in database
Quotation part is updated.Record has the storage address information of data storage wherein in the data directory file.Need access (or
Search) some data when, the data directory file can be passed through and realize the quick angle for accessing (or lookup), being realized from data search
From the point of view of, data directory file can be regarded as other one class file/record, and it, which is included, can indicate that related data record
Various records.Wherein, each index has a corresponding searching code, and any one subset of character field can form one
Individual searching code；So, data directory file is equivalent to a set of all data directory items, and it can be set searching code
All data directory items being worth provide the various effectively supports needed for positioning.Can be with records store data in data directory file
The importing time, to facilitate staff to be managed data storage；It can also be recorded in certain data directory file other
Related import information, such as data source address, raw data format etc..
Reference picture 5, Fig. 5 are the schematic flow sheet of data storage method fourth embodiment of the present invention.
Based on above-mentioned Fig. 2 into Fig. 4 either figure illustrated embodiment, in the present embodiment, after step S30, in addition to：
Step S60, when receiving data call request, the data call that the database is distributed to data call end connects
Mouthful, so that the data call end carries out data call.
In the present embodiment, when initial data is converted into data storage importing database, number is called for convenience of other systems
According to the data storage in storehouse, also calling interface can be distributed to data call end, so that the data call end is entered by the interface
Row data call.Specifically, when receiving external system and sending data call request, by external system allocation database
Data call interface API, and the script file that external system sends is received by interface API, wherein, the API definition is outer
Rule (standard) during portion's system edit script file, and script file is then used to illustrate (description) call operation；Receiving
During script file, you can script file is parsed, so that it is determined that data manipulation instruction corresponding to script file (including operation
Object), and operated according to the data manipulation instruction, to obtain calling data；This can be adjusted when obtaining and calling data
With data feedback to external system, so as to complete the process of data call.
Further, it is contemplated that include private data in database in data storage, in order to prevent these private datas
Used by illegal caller, when receiving data call request, the identity in first can also being asked according to data call
The identity at data call end is verified；If the identity at data call end has passed through checking, then it is assumed that data call end is
It is believable, now can be to the data call interface of data call end allocation database, so that it carries out data call；And if number
By authentication, then its data call request can not be refused, so as to improve the peace of data storage in database according to end is called
Quan Xing.
In addition, the present invention also provides a kind of data loading device.
Reference picture 6, Fig. 6 are the high-level schematic functional block diagram of data loading device first embodiment of the present invention.
In the present embodiment, the data loading device includes：
Model generation module 10, there are the input sample data of same data structure and corresponding with data source for obtaining
Database template data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；
Data input module 20, inputted for obtaining the initial data of the data source, and by the initial data to institute
State Data Translation model；
Data import modul 30, for being converted to the initial data of the data source pair by the Data Translation model
The data storage answered, and the data storage is imported into the database.
Further, the model generation module 10, it is additionally operable to the input sample data and database template respectively
Data are learnt as input training set and output training set, and based on the input training set and output training set to initial machine
Model carries out analyzing and training, and construction obtains Data Translation model, wherein the Data Translation model includes the input sample number
According to the conversion logic between template data.
Further, the data storage includes data type mark, and the data loading device also includes：
Data combination module, for searching similar data storage in the database according to data type mark,
And the similar data storage is subjected to geometric operation, assemblage characteristic data are obtained, to be extended to data dimension.
Further, the data loading device also includes：
Index upgrade module, the data directory file for the importing situation according to the data storage to the database
It is updated.
Further, the data loading device also includes：
Data call module, for when receiving data call request, the database to be distributed to data call end
Data call interface, so that the data call end carries out data call.
Further, the data call module includes：
File obtaining unit, for when receiving data call request, the database to be distributed to data call end
Data call interface, and the script file sent by the data call interface acquisition data call end；
Data transmission unit, for the call operation according to corresponding to script file execution, with the database
Obtain and call data, and the calling data are sent to the data.
Further, the data call module also includes：
Identity authenticating unit, for receive data call request when, according to the data call ask in identity
Identify and the identity at the data call end is verified；
Interface assignment unit, for when the identity at the data call end passes through checking, institute to be distributed to data call end
The data call interface of database is stated, so that the data call end carries out data call.
Wherein, modules are corresponding with each step in above-mentioned data storage method embodiment in data loading device, its
Function and implementation process no longer repeat one by one here.
In addition, the present invention also provides a kind of computer-readable recording medium.
Data loading program is stored with computer-readable recording medium of the present invention, wherein the data loading program is located
When managing device execution, following step is realized：
Obtain has the input sample data of same data structure and corresponding database template data, and root with data source
According to Data Translation model corresponding to the input sample data and the generation of database template data；
The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；
The initial data of the data source is converted to by corresponding data storage by the Data Translation model, and by institute
State data storage and import the database.
Further, the data conversion mould according to corresponding to the input sample data and the generation of database template data
The step of type, includes：
The input sample data and database template data as input training set and are exported into training set respectively, and base
Analyzing and training is carried out to initial machine learning model in the input training set and output training set, construction obtains data conversion mould
Type, wherein the Data Translation model includes conversion logic between the input sample data and template data.
Further, the data storage includes data type mark, and the data loading program is executed by processor
When, also realize following step：
Similar data storage is searched according to data type mark in the database, and by the similar storage number
According to geometric operation is carried out, assemblage characteristic data are obtained, to be extended to data dimension.
Further, when the data loading program is executed by processor, following step is also realized：
The data directory file of the database is updated according to the importing situation of the data storage.
Further, when the data loading program is executed by processor, following step is also realized：
When receiving data call request, the data call interface of the database is distributed to data call end, for
The data call end carries out data call.
Further, it is described when receiving data call request, the data of the database are distributed to data call end
Calling interface, so that the step of data call end progress data call includes：
When receiving data call request, the data call interface of the database is distributed to data call end, and lead to
Cross the data call interface and obtain the script file that the data call end is sent；
The call operation according to corresponding to performing the script file, to obtain calling data in the database, and will
The calling data are sent to the data.
Further, it is described when receiving data call request, the data of the database are distributed to data call end
Calling interface, so that the step of data call end progress data call includes：
When receiving data call request, the identity in being asked according to the data call is to the data call
The identity at end is verified；
When the identity at the data call end passes through checking, the data call of the database is distributed to data call end
Interface, so that the data call end carries out data call.
Wherein, the method realized when data loading program is performed can refer to each reality of data storage method of the present invention
Example is applied, here is omitted.
It should be noted that herein, term " comprising ", "comprising" or its any other variant are intended to non-row
His property includes, so that process, method, article or system including a series of elements not only include those key elements, and
And also include the other element being not expressly set out, or also include for this process, method, article or system institute inherently
Key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that including this
Other identical element also be present in the process of key element, method, article or system.
The embodiments of the present invention are for illustration only, do not represent the quality of embodiment.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can add the mode of required general hardware platform to realize by software, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on such understanding, technical scheme is substantially done to prior art in other words
Going out the part of contribution can be embodied in the form of software product, and the computer software product is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disc, CD), including some instructions to cause a station terminal equipment (can be mobile phone,
Computer, server, air conditioner, or network equipment etc.) perform method described in each embodiment of the present invention.
The preferred embodiments of the present invention are these are only, are not intended to limit the scope of the invention, it is every to utilize this hair
The equivalent structure or equivalent flow conversion that bright specification and accompanying drawing content are made, or directly or indirectly it is used in other related skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data storage method, it is characterised in that the data storage method comprises the following steps：
Obtain has the input sample data of same data structure and corresponding database template data with data source, and according to institute
State Data Translation model corresponding to input sample data and the generation of database template data；
The initial data of the data source is obtained, and the initial data is inputted to the Data Translation model；
The initial data of the data source is converted to by corresponding data storage by the Data Translation model, and deposited described
Store up data and import the database.
2. data storage method as claimed in claim 1, it is characterised in that described according to the input sample data and data
Include corresponding to the generation of library template data the step of Data Translation model：
Respectively using the input sample data and database template data as input training set and output training set, and it is based on institute
To state input training set and output training set and analyzing and training is carried out to initial machine learning model, construction obtains Data Translation model,
Wherein described Data Translation model includes conversion logic between the input sample data and template data.
3. data storage method as claimed in claim 1, it is characterised in that the data storage includes data type mark
Know,
It is described that the initial data of the data source is converted to by corresponding data storage by the Data Translation model, and by institute
After stating the step of data storage imports the database, in addition to：
Similar data storage is searched according to data type mark in the database, and the similar data storage is entered
Row geometric operation, assemblage characteristic data are obtained, to be extended to data dimension.
4. data storage method as claimed in claim 1, it is characterised in that described by described in Data Translation model general
The initial data of data source is converted to corresponding data storage, and the step of the data storage is imported into the database it
Afterwards, in addition to：
The data directory file of the database is updated according to the importing situation of the data storage.
5. the data storage method as any one of Claims 1-4, it is characterised in that described to be turned by the data
The initial data of the data source is converted to corresponding data storage by mold changing type, and the data storage is imported into the data
After the step of storehouse, in addition to：
When receiving data call request, the data call interface of the database is distributed to data call end, for described
Data call end carries out data call.
6. data storage method as claimed in claim 5, it is characterised in that it is described when receiving data call request, to
The data call interface of the database is distributed at data call end, for bag the step of the progress data call of the data call end
Include：
When receiving data call request, the data call interface of the database is distributed to data call end, and pass through institute
State data call interface and obtain the script file that the data call end is sent；
The call operation according to corresponding to performing the script file, to obtain calling data in the database, and by described in
Data are called to send to the data call end.
7. data storage method as claimed in claim 5, it is characterised in that it is described when receiving data call request, to
The data call interface of the database is distributed at data call end, for bag the step of the progress data call of the data call end
Include：
When receiving data call request, the identity in being asked according to the data call is to the data call end
Identity is verified；
When the identity at the data call end passes through checking, the data call that the database is distributed to data call end connects
Mouthful, so that the data call end carries out data call.
8. a kind of data loading device, it is characterised in that the data loading device includes：
Model generation module, there are the input sample data of same data structure and corresponding database with data source for obtaining
Template data, and the Data Translation model according to corresponding to the input sample data and the generation of database template data；
Data input module, inputted for obtaining the initial data of the data source, and by the initial data to the data
Transformation model；
Data import modul, for the initial data of the data source to be converted into corresponding deposit by the Data Translation model
Data are stored up, and the data storage is imported into the database.
9. a kind of data loading equipment, it is characterised in that the data loading equipment includes processor, memory and storage
And can be by the data loading program of the computing device, wherein the data loading program is by the place on the memory
When managing device and performing, the step of realizing data storage method as any one of claim 1 to 7.
10. a kind of computer-readable recording medium, it is characterised in that data loading journey is stored with the readable storage medium storing program for executing
Sequence, wherein when the data loading program is executed by processor, realize that the data as any one of claim 1 to 7 enter
The step of storehouse method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710873319.9A CN107679141A (en) | 2017-09-25 | 2017-09-25 | Data storage method, device, equipment and computer-readable recording medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710873319.9A CN107679141A (en) | 2017-09-25 | 2017-09-25 | Data storage method, device, equipment and computer-readable recording medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107679141A true CN107679141A (en) | 2018-02-09 |
Family
ID=61137174
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710873319.9A Pending CN107679141A (en) | 2017-09-25 | 2017-09-25 | Data storage method, device, equipment and computer-readable recording medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107679141A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108647304A (en) * | 2018-05-09 | 2018-10-12 | 中国平安人寿保险股份有限公司 | Report display system, method, computer equipment and storage medium |
| CN109033452A (en) * | 2018-08-23 | 2018-12-18 | 重庆富民银行股份有限公司 | A kind of data warehouse is intelligent to construct stowage and system |
| CN109062875A (en) * | 2018-07-06 | 2018-12-21 | 广东省智能制造研究所 | A kind of end C/S three-dimensional modeling data conversion method, device and storage medium |
| CN109657413A (en) * | 2019-01-23 | 2019-04-19 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN109977162A (en) * | 2019-04-10 | 2019-07-05 | 广东省城乡规划设计研究院 | A kind of urban and rural planning data transfer device, system and computer readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092980A (en) * | 2013-01-31 | 2013-05-08 | 中国科学院自动化研究所 | Method and system of data automatic conversion and storage |
| CN104361221A (en) * | 2014-10-31 | 2015-02-18 | 沈阳锐易特软件技术有限公司 | Heterogeneous system data mapping template-based medical data acquisition system and method |
| US20160162557A1 (en) * | 2014-12-03 | 2016-06-09 | Sas Institute Inc. | System to convert semantic layer metadata to support database conversion |
| CN106066892A (en) * | 2016-06-20 | 2016-11-02 | 四川上略互动网络技术有限公司 | A kind of travel information data processing method based on multisource data fusion |
| CN106682235A (en) * | 2017-01-18 | 2017-05-17 | 济南浪潮高新科技投资发展有限公司 | System and method for isomerous data mapping |
| CN107193858A (en) * | 2017-03-28 | 2017-09-22 | 福州金瑞迪软件技术有限公司 | Towards the intelligent Service application platform and method of multi-source heterogeneous data fusion |
-
2017
- 2017-09-25 CN CN201710873319.9A patent/CN107679141A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092980A (en) * | 2013-01-31 | 2013-05-08 | 中国科学院自动化研究所 | Method and system of data automatic conversion and storage |
| CN104361221A (en) * | 2014-10-31 | 2015-02-18 | 沈阳锐易特软件技术有限公司 | Heterogeneous system data mapping template-based medical data acquisition system and method |
| US20160162557A1 (en) * | 2014-12-03 | 2016-06-09 | Sas Institute Inc. | System to convert semantic layer metadata to support database conversion |
| CN106066892A (en) * | 2016-06-20 | 2016-11-02 | 四川上略互动网络技术有限公司 | A kind of travel information data processing method based on multisource data fusion |
| CN106682235A (en) * | 2017-01-18 | 2017-05-17 | 济南浪潮高新科技投资发展有限公司 | System and method for isomerous data mapping |
| CN107193858A (en) * | 2017-03-28 | 2017-09-22 | 福州金瑞迪软件技术有限公司 | Towards the intelligent Service application platform and method of multi-source heterogeneous data fusion |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108647304A (en) * | 2018-05-09 | 2018-10-12 | 中国平安人寿保险股份有限公司 | Report display system, method, computer equipment and storage medium |
| CN108647304B (en) * | 2018-05-09 | 2023-11-21 | 中国平安人寿保险股份有限公司 | Report display system, method, computer device and storage medium |
| CN109062875A (en) * | 2018-07-06 | 2018-12-21 | 广东省智能制造研究所 | A kind of end C/S three-dimensional modeling data conversion method, device and storage medium |
| CN109033452A (en) * | 2018-08-23 | 2018-12-18 | 重庆富民银行股份有限公司 | A kind of data warehouse is intelligent to construct stowage and system |
| CN109033452B (en) * | 2018-08-23 | 2021-09-07 | 重庆富民银行股份有限公司 | Intelligent construction loading method and system for data warehouse |
| CN109657413A (en) * | 2019-01-23 | 2019-04-19 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN109657413B (en) * | 2019-01-23 | 2023-05-23 | 中国人民解放军国防科技大学 | Simulation component model warehousing method, model library, computer equipment and storage medium |
| CN109977162A (en) * | 2019-04-10 | 2019-07-05 | 广东省城乡规划设计研究院 | A kind of urban and rural planning data transfer device, system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wu et al. | Smart cities in Taiwan: A perspective on big data applications | |
| US20200110792A1 (en) | Customized customer relationship management platform method and devices | |
| CN107679141A (en) | Data storage method, device, equipment and computer-readable recording medium | |
| CN109684530B (en) | Information push service system based on web management and mobile phone applet application | |
| CN109344223A (en) | Building information model management system and method based on cloud computing technology | |
| CN108509339A (en) | Method for generating test case, device based on browser and mind map and equipment | |
| CN111061756B (en) | Data platform, data processing method and electronic equipment | |
| CN112215171B (en) | Target detection method, device, equipment and computer readable storage medium | |
| CN107832468A (en) | Demand recognition methods and device | |
| CN113570030B (en) | Data processing method, device, equipment and storage medium | |
| CN106484766B (en) | Searching method and device based on artificial intelligence | |
| CN107861954A (en) | Information output method and device based on artificial intelligence | |
| CN105824855B (en) | Method and device for screening and classifying data objects and electronic equipment | |
| CN110020358A (en) | Method and apparatus for generating dynamic page | |
| CN107463935A (en) | Application class methods and applications sorter | |
| US20190385472A1 (en) | Assessment item generation and scoring | |
| CN113822521B (en) | Method, device and storage medium for detecting quality of question library questions | |
| US10275462B2 (en) | Automatic translation of string collections | |
| CN116720489B (en) | Page filling method and device, electronic equipment and computer readable storage medium | |
| CN108388811A (en) | Personalized study under wechat public platform | |
| CN107451194A (en) | A kind of image searching method and device | |
| CN117194620A (en) | Information processing method, device, equipment and storage medium | |
| CN116975254A (en) | Visual data presentation method, visual data presentation device and storage medium | |
| Fain et al. | Enterprise web development: Building HTML5 applications: from desktop to mobile | |
| CN116739845A (en) | Contract generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180604 Address after: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant after: Shenzhen one ledger Intelligent Technology Co., Ltd. Address before: 200000 Xuhui District, Shanghai Kai Bin Road 166, 9, 10 level. Applicant before: Shanghai Financial Technologies Ltd |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180209 |
|
| RJ01 | Rejection of invention patent application after publication |