CN103970914B - A kind of acquisition and storage method for isomeric data between sewage treatment plant - Google Patents
A kind of acquisition and storage method for isomeric data between sewage treatment plant Download PDFInfo
- Publication number
- CN103970914B CN103970914B CN201410175754.0A CN201410175754A CN103970914B CN 103970914 B CN103970914 B CN 103970914B CN 201410175754 A CN201410175754 A CN 201410175754A CN 103970914 B CN103970914 B CN 103970914B
- Authority
- CN
- China
- Prior art keywords
- data
- dictionary
- treatment plant
- sewage treatment
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
It is used for the acquisition and storage method of isomeric data between sewage treatment plant the present invention relates to a kind of, solves that isomeric data can not be stored, transmission bandwidth can not meet the defect of needs compared with prior art.The present invention comprises the following steps：Define data dictionary；Called data；Data are converted；Data file is stored.The present invention reduces sewage treatment plant's reported data bandwidth consumption, effective integration Duo Jia sewage treatment plants isomeric data, and solves the storage problem of mass data.
Description
Technical field
It is specifically a kind of to be used for isomery number between sewage treatment plant the present invention relates to data acquisition technical field of memory
According to acquisition and storage method.
Background technology
Sewage treatment industry is generally relatively low due to causing initial stage " reconstruction set, gently run " sewage disposal to run rate of load condensate,
Under the guidance of 12 " energy-saving and emission-reduction " policies, sewage treatment industry builds sewage treatment plant by original batch, turns to lifting
The treatment effeciency and managerial skills of sewage treatment plant.Then sewage treatment plant starts to build central control system, is adopted by mass data
Collection and Automated condtrol, perform lean operation management.But row, which omits in printing, water treatment is not up to standard etc. disobeys is stolen to sewage treatment plant
Rule behavior still can not be supervised effectively, and sewage treatment plant can not efficiently manage under water utilities company pair, it is therefore desirable to many dirts
The gathered data of water treatment plant collects together, is available for inquiry and analysis.
By the central control system of sewage treatment plant, can with the operation on taken at regular intervals to sewage disposal each process node and
Monitoring Data, then the gathered data of multiple sewage treatment plants can just be gathered by data acquisition gateway.But need
Collect storing from multiple sewage treatment plant's gathered datas and then encounter problems with：
1st, the information of collection point is generally comprised in gathered data, and the collection point information of Ge Jia treatment plants is different, data tool
There is isomerism, data storage can not be collected after collecting；
2nd, the frequency of gathered data generally is second level, and reported data is difficult to very big redundancy, transmission bandwidth demand
It is met；
3rd, sewage treatment plant needs to retain historical data for many years, and the data after collecting are magnanimity, the data after collecting
It is then extremely difficult that storehouse wants to look up data.
How to develop a kind of data processing method stored between isomeric data that can meet and have become urgent need solution
Technical problem.
The content of the invention
The invention aims to solve, isomeric data in the prior art can not be stored, transmission bandwidth can not meet needs
Defect solved the above problems there is provided a kind of acquisition and storage method for being used for isomeric data between sewage treatment plant.
To achieve these goals, technical scheme is as follows：
A kind of acquisition and storage method for isomeric data between sewage treatment plant, comprises the following steps：
Data dictionary is defined, Ge Jia sewage treatment plants define the data dictionary of oneself；
Called data, data access module obtains sewage treatment plant's field data from multiple sewage treatment plants, is stored in data
In buffer pool；
Data are converted, and Data buffer periodically writes the data of data access module as the self-explanatory number comprising data dictionary
According to file；
Data file is stored, and Data buffer periodically parses self-explanatory data file, data dictionary and gathered data are deposited
Store up in NoSQL databases.
Described definition data dictionary comprises the following steps：
Treatment plant's dictionary information in each sewage treatment plant's difference log-on data dictionary, including treatment plant's Name ＆ Location letter
Breath, generates unique treatment plant's mark；
Data dictionary, including technological process dictionary, equipment dictionary, collection point dictionary, technological process are defined using csv file
Dictionary, equipment dictionary, collection point dictionary are constituted by gauge outfit and example；Gauge outfit includes necessary field and extended field, necessary word
The dictionary information that section must provide for each treatment plant, extended field is according to the increased information of actual conditions；Wherein：
Technological process dictionary is the definition to treatment plant's sewage treatment process, and the necessary field that gauge outfit is included includes mark ID
With the title of flow, example is each nodal information for constituting sewage treatment plant's handling process；
Equipment dictionary is the definition to various equipment in treatment plant, and the necessary field that gauge outfit is included includes mark ID, equipment
Process node belonging to name and equipment, example is the various equipment in treatment plant；
Collection point dictionary is the definition to data collection point, and the necessary field that gauge outfit is included includes mark ID, collection point and retouched
State, the field that collection point is shown in form, generation form when calculation formula, the type of gathered data, the unit of data, data
Maximin, equipment corresponding device information and the virtual point data of normal range (NR), virtual point data are fixed with virtual_ prefixes
Justice；
Data dictionary csv file is associated with treatment plant dictionary information, set up treatment plant's title and respective handling factory number
According to the correspondence of dictionary.
Described called data comprises the following steps：
Data are reported interface to be supplied to Ge Jia sewage treatment plants by data access module in the way of web server；
Reported data is set up in sewage treatment plant, and reported data includes the mark ID and gathered data for the treatment of plant, gathered data
Include acquisition time, collection point identification ID, collection numerical value, treatment plant mark ID and collection point identification ID, from the number for the treatment of plant
ID and collection point identification ID are identified according to corresponding treatment plant is transferred in dictionary；
Reported data is encoded using Json forms and sent to Data buffer, and its data format is as follows：
factory_id：Obtained treatment plant is registered from data dictionary module and identifies ID；data：Array set；Data bags
Include data below form：datatime：The time of signaling point collection；signal_id：The signaling point defined in data dictionary module
Identify ID；value：Signal point values.
Described data conversion comprises the following steps：
Data buffer receives the reported data of Json forms using FIFO mechanism, and the data read out are deleted from buffer pool
Remove and be transmitted to memory module, be processed into self-explanatory data file, the time when filename of self-explanatory data file is to read
Stamp name；
Self-explanatory data file is periodically generated by Data buffer, self-explanatory data file includes dictionary portion and data
Part, treatment plant's dictionary information is added to before dictionary portion, will processing in technological process ID, device id, collection point ID
Factory ID be added in before as prefix；The gauge outfit of data division is collection point ID, acquisition time, collection numerical value, before the ID of collection point
Treatment plant ID is added as prefix in face, and collection numerical value is the interior all gathered datas preserved of process cycle in buffer pool.
Described data file storage comprises the following steps：
Set the process cycle of memory module consistent with the process cycle of Data buffer；
NoSQL databases design, NoSQL database defined in data dictionary table sum consistent with self-explanatory data file
According to table, the storage organization of NoSQL databases is defined as key-value pairs towards row, each row of data composition structure is：
Rowkey, column_name, column_value,
Wherein：
In the storage organization of data dictionary table, rowkey is mark ID, column_name in document time stamp+dictionary
It is the value of correspondence ID attribute for other attribute-names in dictionary, column_value；
In the storage organization of tables of data, rowkey be signaling point ID+ acquisition times stamp, column_name be svalue or
Dictionary_version, when column_name is svalue, column_value is correspondence signaling point ID collection number
Value, when column_name is dictionary_version, column_value is corresponding to rowkey in data dictionary table
Timestamp；
Definition according to data dictionary table and tables of data stores dictionary portion and data division respectively, by memory module
In data Cun Chudao NoSQL databases.
Also include the not processed judgement into self-explanatory data file of data file, it comprises the following steps：
In the data dictionary table of NoSQL databases, timestamp maximum in rowkey is obtained, all data files are traveled through
Filename, if timestamp in filename is more than timestamp maximum in database, then it represents that the not processed mistake of this document；
The file of not processed mistake is handled one by one, parses data file, dictionary portion and data division are stored respectively.
Beneficial effect
A kind of acquisition and storage method for isomeric data between sewage treatment plant of the present invention, subtracts compared with prior art
The reported data bandwidth consumption of few sewage treatment plant, effective integration Duo Jia sewage treatment plants isomeric data, and solve mass data
Storage problem.By the configuration of data dictionary, specification defines the collection point information of sewage treatment plant.By the way that collection point is belonged to
Property information and data separating, reduce the complexity that data are reported.By adding data dictionary in the data file, make file
With self-explanatory, it is ensured that data are complete in time attribute, historical data is allowed to keep associating with corresponding historical status, very
Restoring data scene, increases the reliability of historical data analysis on the spot.
Brief description of the drawings
Fig. 1 is flow chart of the method for the present invention
Embodiment
To make to have a better understanding and awareness to architectural feature of the invention and the effect reached, to preferably
Embodiment and accompanying drawing coordinate detailed description, are described as follows：
As shown in figure 1, a kind of acquisition and storage method for isomeric data between sewage treatment plant of the present invention, bag
Include following steps：
The first step, defines data dictionary, and Ge Jia sewage treatment plants define the data dictionary of oneself.Each sewage treatment plant's note
Volume succeeds after the information of oneself, generates unique treatment plant's mark ID, and the data dictionary configured afterwards is associated with our factory ID.Number
The explanation and illustration to gathered data according to dictionary, define it is each collection point data attribute, including its corresponding device information,
Technological process information and sewage treatment plant's information.It is comprised the following steps that:
(1）Treatment plant's dictionary information in each sewage treatment plant's difference log-on data dictionary, including treatment plant's title
（name）And positional information（location）, generate unique treatment plant's mark（factory_id）.Treatment plant's dictionary information is
Each corresponding sewage treatment plant's information, unique treatment plant's determination is carried out by unique treatment plant mark.
（2）Data dictionary is defined using csv file, according to data dictionary, ID letters in collection point need to be only carried in reported data
Breath, is to belong in which the platform equipment in which technological process of which treatment plant, collection with regard to that can inquire a reported data
Point, which represents what meaning, data format and unit, is what, data normal range (NR) are how many, how to calculate and shown in form
The range of information such as value, so each reported data, which is given tacit consent to, has taken many explanatory information, it is convenient in generation form and
Unscrambling data during data analysis.Data dictionary also includes technological process dictionary, equipment dictionary, collection point dictionary.Technological process word
Allusion quotation, equipment dictionary and collection point dictionary are constituted by gauge outfit and example；Gauge outfit includes necessary field and extended field, and English is used respectively
Word segment table shows that first character section is fixed as identifying ID.Necessary field is the dictionary information that each treatment plant must provide, and is expanded
It is according to the increased information of actual conditions to open up field；When sewage treatment plant needs to sort out some gathered datas to enter line number
During according to analysis or technique upgrading, the extended field of oneself can be defined.Example be treatment plant in constitute the specific of the dictionary
One point, is distinguished by identifying ID.Each several part dictionary is separated by line Separator.
Wherein：
Technological process dictionary is the definition to treatment plant's sewage treatment process, and the necessary field that gauge outfit is included includes mark ID
process_id（Identified in each treatment plant unique）With the title process_name of flow（Technological process nodename）,
Example is each nodal information of composition sewage treatment plant's handling process, such as coarse rack, fine fack, aeration tank etc..
Equipment dictionary is the definition to various equipment in treatment plant, and the necessary field that gauge outfit is included includes mark ID
device_id（Identified in each treatment plant unique）, device name device_name and the process node belonging to equipment
process_id（It can be sky, that is, be not belonging to any technological process node）, example is the various equipment in treatment plant, such as rouses
Blower fan, elevator pump etc..
Collection point dictionary is the definition to data collection point, and the necessary field that gauge outfit is included includes mark ID, collection point and retouched
State, the field that collection point is shown in form, generation form when calculation formula, the type of gathered data, the unit of data, data
Maximin, equipment corresponding device information and the virtual point data of normal range (NR), virtual point data are fixed with virtual_ prefixes
Justice.Dictionary necessary field in collection point is more, as follows：
signal_id：ID is identified, is identified in each treatment plant unique.Identify and use for virtual signal
" virtual_ " field adds numerical identity, such as：“virtual_001”.Manual signal is identified using " manual_ ", such as：
“manual_001”。
signal_desc：Collection point is described.
signal_mark：The field that collection point is shown in form.
data_type：The type of gathered data, can be integer, floating type, 0/1 semaphore etc..
unit：The unit of data.
calc_expr：Generate calculation formula during form.
min：The minimum value of the lowest critical point of collection point early warning value, i.e. data normal range (NR).
max：The maximum of the maximum critical point of collection point early warning value, i.e. data normal range (NR).
device_id：Equipment corresponding device information, can be sky.
Collection point dictionary is except the point reported in an automated manner of physical presence, also comprising virtual by manually reporting
Some chemical examination, administration data points, can make a distinction, such as virtual point is fixed using virtual_ prefixes during definition in mark ID
Justice.
It is the example of one data word allusion quotation csv file below：
## technological process dictionary portions
process_id,process_name
Proc_001, coarse rack
……
------------（Dictionary separator）
## equipment dictionary portions
device_id,device_name,process_id
Dev_001, left coarse rack, proc_001
……
------------（Dictionary separator）
## collection points dictionary portion
signal_id,signal_desc,signal_mark,data_type,unit,calc_expr,min,max,
device_id
Virtual_001, intake PH, PH_IN, float, nothing, avg, 5.0,11.0, proc_001, dev_001
……
The pattern of wherein dictionary separator is only used as example, can arbitrarily define.
（3）Data dictionary csv file is associated with treatment plant dictionary information, set up treatment plant's title and respective handling factory
The correspondence of data dictionary.Csv file is associated with treatment plant ID so that can inquire alignment processing factory according to factory_id
Data dictionary.
Second step, called data, data access module obtains sewage treatment plant's field data from multiple sewage treatment plants, deposits
Enter in Data buffer.It is comprised the following steps that：
（1）Data are reported interface to be supplied to Ge Jia sewage treatment plants by data access module in the way of web server.
Due to there are multiple sewage treatment plants, then there is the select permeability of interface, entered using web server modes of the prior art
OK.When needing selection A factories, data are then reported interface to be supplied to A factories by data access module, are now set up A factories and are connect with data
Enter the data communication between module；When if desired selecting B factories, data are then reported interface to be supplied to B factories by data access module, this
Data communication between Shi Jianli B factories and data access module.
（2）Reported data is set up in sewage treatment plant, and reported data includes the mark ID and gathered data for the treatment of plant, gathers number
According to acquisition time, collection point identification ID, collection numerical value, treatment plant mark ID and collection point identification ID is included, from treatment plant
Corresponding treatment plant mark ID and collection point identification ID are transferred in data dictionary.
（3）Reported data is encoded using Json forms and sent to Data buffer, and its data format is as follows：
factory_id：Obtained treatment plant is registered from data dictionary module and identifies ID；data：Array set；
Collection in worksite numerical value is dictionary set, and form is { " datetime ":“yyyy-mm-dd hh:mm:ss”,
“signal_id”:“001”,“value”:" 123.3 " }, wherein
datatime：The time of signaling point collection；signal_id：The signal point identification ID defined in data dictionary module；
value：Signal point values.
Example is as follows：
[{"factory_id":"jnshw_01","data":[{"datetime":"yyyy-mm-dd hh:mm:ss","
signal_id":"auto_001","value":"123.3"}, {"datetime":"yyyy-mm-dd hh:mm:
ss.zzz","signal_id":"auto_002","value":"123.3"},{"datetime":"yyyy-mm-dd hh:
mm:ss.zzz","signal_id":"manual_001","value":"123.3"}]}]。
3rd step, data conversion, Data buffer is periodically write the data of data access module comprising data dictionary as
Self-explanatory data file.It is comprised the following steps that：
（1）Data buffer uses FIFO（FIFO）Mechanism receives the reported data of Json forms, the number read out
According to being deleted from buffer pool and being transmitted to memory module, self-explanatory data file, the filename of self-explanatory data file are processed into
Timestamp name during reading.Explain that data file effect is to be matched dictionary portion and data division, data division
Matched one by one with the dictionary portion corresponding to it.Because each treatment plant's self-defining data dictionary, if data dictionary
Definition is not matched with data division, then can not correctly explain the content of data division.The filename of self-explanatory data file is to read
When taking timestamp name effect be in order to judge whether data file not processed into self-explanatory data file in next step,
The self-explanatory data file of generation is named with the timestamp of start to process, the similar 20140401102000.csv of form, below for
Example：
## treatment plants dictionary portion
factory_id,factory_name,factory_location
Fac_001, the first treatment plant, x areas of x cities x roads x
……
------------（Dictionary separator）
## technological process dictionary portions
process_id,process_name
Fac_001-proc_001, coarse rack
……
------------（Dictionary separator）
## equipment dictionary portions
device_id,device_name,process_id
Fac_001-dev_001, left coarse rack, fac_001-proc_001
……
------------（Dictionary separator）
## collection points dictionary portion
signal_id,signal_desc,signal_mark,data_type,unit,calc_expr,min,max,
device_id
Fac_001-virtual_001, intake PH, PH_IN, float, nothing, avg, 5.0,11.0, fac_001-dev_
001
……
------------（Dictionary separator）
## actual acquired datas part
signal_id,datetime,value
fac_001-virtual_001, 2014-03-31 23:00:10,9.0
fac_001-virtual_001, 2014-03-31 23:00:20,9.0
……
（2）Self-explanatory data file is periodically generated by Data buffer, Data buffer is first received according to FIFO mechanism
To data be first read.Buffer pool sets the cycle of digital independent, depending on according to its size and the quantity of access treatment plant
Phase generates oneself instrument of interpretation.Cycle is unsuitable oversize, because the data crossed before long period may result in are capped, typically with
Hour is period treatment, if data volume less can also be one day etc..Self-explanatory data file includes dictionary portion and data portion
Point, in dictionary portion, treatment plant's dictionary information is added to before dictionary portion, in technological process ID, device id, collection point
In ID by treatment plant ID be added in before as prefix, form global unique mark.In data division, the gauge outfit of data division
For collection point ID, acquisition time, collection numerical value, treatment plant ID is added before the ID of collection point as prefix, the overall situation is formed only
One mark.It is the interior all gathered datas preserved of process cycle, i.e., one hour in buffer pool to gather numerical value（One day）It is interior to receive
All gathered datas arrived.
4th step, data file storage, memory module periodically parses self-explanatory data file, by data dictionary and collection number
According to storage into NoSQL databases.
After self-explanatory data file is resolved, it is saved in NoSQL databases and is used for inquiry.The self-explanatory number being parsed
Release memory space can be directly deleted according to file, the file system of backup can also be saved in.NoSQL databases are NoSQL
, structured database towards row.NoSQL databases data storage in the form of a table, table is made up of row and column, and row is by row master
Key rowkey, attribute column_name and property value column_value compositions, are all preserved with character string forms.Data structure
It is as shown in table 1 below：
| rowkey | column_name | column_value |
| rowkey_001 | column_001 | value_101 |
| rowkey_001 | column_002 | value_102 |
| rowkey_002 | column_001 | value_201 |
| rowkey_002 | column_003 | value_301 |
The data structure of the NoSQL databases of table 1
Row is made up of rowkey+ column_name, so difference rowkey attribute column_name can be different
（Such as rowkey_002 does not have column_002, there is column_003）, attribute easily extends.Storage physically according to
Rowkey lexcographical order arrangement（Rowkey_001 is before rowkey_002）If what can be used when rowkey is by inquiring about is several
Dominant query conditional combination, it is possible to allow the row often read together to be stored together, with efficient query performance.For example make
Rowkey is constituted with [factory_id] _ [signal_id] _ [time] mode, then the data of same treatment plant are just protected
In the presence of together, the data of same signaling point are saved together in same treatment plant, the data of same signaling point be again by
According to time sequencing arrangement, to inquire about data of some signaling point of some factory within certain time will efficiency it is very high.
It specifically includes following steps：
（1）The process cycle of setting memory module is consistent with the process cycle of Data buffer, and Data buffer is typically set
For a hour, the cycle of processing module can also be more than, such as using day as the cycle, because sewage treatment industry demand is usually
Went out the form of the previous day at second day.
（2）NoSQL databases design, NoSQL database defined in data dictionary table consistent with self-explanatory data file
And tables of data, the storage organization of NoSQL databases is defined as key-value pairs towards row, each row of data composition structure is：
Rowkey, column_name, column_value,
Wherein：
In the storage organization of data dictionary table, rowkey is mark ID, column_name in document time stamp+dictionary
It is the value of correspondence ID attribute for other attribute-names in dictionary, column_value.Storage organization example is as follows：
| rowkey | column_name | column_value |
| 1397808993fac_001 | factory_name | First treatment plant |
| 1397808993fac_001 | factory_location | X areas of x cities x roads x |
| 1397808993fac_001-proc_001 | process_name | Coarse rack |
| 1397808993fac_001-dev_001 | device_name | Left coarse rack |
The data store organisation of data dictionary table in the NoSQL databases of table 2
In the storage organization of tables of data, rowkey be signaling point ID+ acquisition times stamp, column_name be svalue or
Dictionary_version, when column_name is svalue, column_value is correspondence signaling point ID collection number
Value, when column_name is dictionary_version, column_value is corresponding to rowkey in data dictionary table
Timestamp.Storage organization example is as follows：
| rowkey | column_name | column_value |
| fac_001-virtual_001_1397804400 | svalue | 9.0 |
| fac_001-virtual_001_1397804400 | dictionary_version | 1397808993 |
| fac_001-virtual_001_1397804410 | svalue | 9.0 |
| fac_001-virtual_001_1397804410 | dictionary_version | 1397808993 |
The data store organisation of tables of data in the NoSQL databases of table 3
Because rowkey is arranged according to lexcographical order, the character length that ensure each rowkey is consistent.
ID and timestamp can determine a maximum length, failing to maximum length is reached, can be with " 0 " polishing.
（3）Definition according to data dictionary table and tables of data stores dictionary portion and data division respectively, by memory module
In data Cun Chudao NoSQL databases in, complete data storage.
In order to ensure the accuracy of data storage, data file is prevented not processed into self-explanatory data file situation
Appearance, the not processed judgement into self-explanatory data file of data file can also be included.
（4）In the data dictionary table of NoSQL databases, timestamp maximum in rowkey is obtained, all data are traveled through
The filename of file, if the timestamp in filename is more than timestamp maximum in database, then it represents that this document is not processed
Cross.What such as in the current database entitled rowkey009 of maximum time stamp file, rowkey009 were then handled well for last
Self-explanatory file, then it is not processed into self-explanatory file more than rowkey009 in the data file.Travel through all data texts
The filename of part, sees if there is the filename more than rowkey009, such as rowkey010, rowkey011, if so, then illustrating
Both of these documents does not carry out the data of self-explanatory file, then handles the file of not processed mistake one by one, data file is parsed, by word
Allusion quotation part and data division are stored respectively, and at this moment maximum time stamp filename is then rowkey011 in current database.
The present invention separates collection point data with collection point attribute information, simplifies data and reports content, makes many sewage
The isomeric data for the treatment of plant can be merged, and can dynamically change collection point without changing in the case that storage logical sum reads logic
Attribute.
General principle, principal character and the advantages of the present invention of the present invention has been shown and described above.The technology of the industry
Personnel are it should be appreciated that the present invention is not limited to the above embodiments, and that described in above-described embodiment and specification is the present invention
Principle, various changes and modifications of the present invention are possible without departing from the spirit and scope of the present invention, these change and
Improvement is both fallen within the range of claimed invention.The protection domain of application claims by appended claims and its
Equivalent is defined.
Claims (4)
1. a kind of acquisition and storage method for isomeric data between sewage treatment plant, it is characterised in that comprise the following steps：
11）Data dictionary is defined, Ge Jia sewage treatment plants define the data dictionary of oneself；
Described definition data dictionary comprises the following steps：
111）Treatment plant's dictionary information in each sewage treatment plant's difference log-on data dictionary, including treatment plant's Name ＆ Location letter
Breath, generates unique treatment plant's mark；
112）Data dictionary, including technological process dictionary, equipment dictionary, collection point dictionary, technological process are defined using csv file
Dictionary, equipment dictionary, collection point dictionary are constituted by gauge outfit and example；Gauge outfit includes necessary field and extended field, necessary word
The dictionary information that section must provide for each treatment plant, extended field is according to the increased information of actual conditions；Wherein：
Technological process dictionary is the definition to treatment plant's sewage treatment process, and the necessary field that gauge outfit is included includes mark ID and stream
The title of journey, example is each nodal information for constituting sewage treatment plant's handling process；
Equipment dictionary is the definition to various equipment in treatment plant, and the necessary field that gauge outfit is included includes mark ID, device name
With the process node belonging to equipment, example is the various equipment in treatment plant；
Collection point dictionary is the definition to data collection point, and the necessary field that gauge outfit is included includes mark ID, collection point description, adopted
Calculation formula, the type of gathered data, the unit of data, data of the collection point in field, the generation form of form displaying are normal
Maximin, equipment corresponding device information and the virtual point data of scope, virtual point data are defined with virtual_ prefixes；
113）Data dictionary csv file is associated with treatment plant dictionary information, set up treatment plant's title and respective handling factory number
According to the correspondence of dictionary；
12）Called data, data access module obtains sewage treatment plant's field data from multiple sewage treatment plants, and deposit data are delayed
Rush in pond；
13）Data are converted, and Data buffer periodically writes the data of data access module as the self-explanatory number comprising data dictionary
According to file；Its following steps：
131）Data buffer receives the reported data of Json forms using FIFO mechanism, and the data read out are deleted from buffer pool
Remove and be transmitted to memory module, be processed into self-explanatory data file, the time when filename of self-explanatory data file is to read
Stamp name；
132）Self-explanatory data file is periodically generated by Data buffer, self-explanatory data file includes dictionary portion and data
Part, treatment plant's dictionary information is added to before dictionary portion, will processing in technological process ID, device id, collection point ID
Factory ID be added in before as prefix；The gauge outfit of data division is collection point ID, acquisition time, collection numerical value, before the ID of collection point
Treatment plant ID is added as prefix in face, and collection numerical value is the interior all gathered datas preserved of process cycle in buffer pool；
14）Data file is stored, and memory module periodically parses self-explanatory data file, data dictionary and acquired data storage are arrived
In NoSQL databases.
2. a kind of acquisition and storage method for isomeric data between sewage treatment plant according to claim 1, its feature
It is that described called data comprises the following steps：
21）Data are reported interface to be supplied to Ge Jia sewage treatment plants by data access module in the way of web server；
22）Reported data is set up in sewage treatment plant, and reported data is included in the mark ID and gathered data for the treatment of plant, gathered data
Including acquisition time, collection point identification ID, collection numerical value, treatment plant mark ID and collection point identification ID, from the data for the treatment of plant
Corresponding treatment plant mark ID and collection point identification ID are transferred in dictionary；
23）Reported data is encoded using Json forms and sent to Data buffer, and its data format is as follows：
factory_id：Obtained treatment plant is registered from data dictionary module and identifies ID；data：Array set；Data include with
Lower data format：datatime：The time of signaling point collection；signal_id：The signal point identification defined in data dictionary module
ID；value：Signal point values.
3. a kind of acquisition and storage method for isomeric data between sewage treatment plant according to claim 1, its feature
It is that described data file storage comprises the following steps：
31）Set the process cycle in memory module consistent with the process cycle of Data buffer；
32）NoSQL databases design, NoSQL database defined in data dictionary table sum consistent with self-explanatory data file
According to table, the storage organization of NoSQL databases is defined as key-value pairs towards row, each row of data composition structure is：
Rowkey, column_name, column_value,
Wherein：
In the storage organization of data dictionary table, rowkey is that mark ID, column_name in document time stamp+dictionary is word
Other attribute-names in allusion quotation, column_value are the values of correspondence ID attribute；
In the storage organization of tables of data, rowkey be signaling point ID+ acquisition times stamp, column_name be svalue or
Dictionary_version, when column_name is svalue, column_value is correspondence signaling point ID collection number
Value, when column_name is dictionary_version, column_value is corresponding to rowkey in data dictionary table
Timestamp；
33）Definition according to data dictionary table and tables of data stores dictionary portion and data division respectively, by memory module
In data Cun Chudao NoSQL databases.
4. a kind of acquisition and storage method for isomeric data between sewage treatment plant according to claim 3, its feature
It is also to include the not processed judgement into self-explanatory data file of data file, it comprises the following steps：
41）In the data dictionary table of NoSQL databases, timestamp maximum in rowkey is obtained, all data files are traveled through
Filename, if timestamp in filename is more than timestamp maximum in database, then it represents that the not processed mistake of this document；
42）The file of not processed mistake is handled one by one, parses data file, dictionary portion and data division are stored respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410175754.0A CN103970914B (en) | 2014-04-29 | 2014-04-29 | A kind of acquisition and storage method for isomeric data between sewage treatment plant |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410175754.0A CN103970914B (en) | 2014-04-29 | 2014-04-29 | A kind of acquisition and storage method for isomeric data between sewage treatment plant |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103970914A CN103970914A (en) | 2014-08-06 |
| CN103970914B true CN103970914B (en) | 2017-08-11 |
Family
ID=51240412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410175754.0A Active CN103970914B (en) | 2014-04-29 | 2014-04-29 | A kind of acquisition and storage method for isomeric data between sewage treatment plant |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103970914B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102014118618A1 (en) * | 2014-12-15 | 2016-06-16 | Endress + Hauser Conducta Gesellschaft für Mess- und Regeltechnik mbH + Co. KG | Method for controlling a process variable |
| CN105808534B (en) * | 2014-12-27 | 2019-06-11 | 株洲中车时代电气股份有限公司 | A kind of method that isomeric data merges extraction initial data in file |
| CN105681285B (en) * | 2015-12-30 | 2018-10-09 | 合肥城市云数据中心股份有限公司 | A kind of isomery industry signal source information acquisition methods |
| CN105701148B (en) * | 2015-12-30 | 2019-02-22 | 合肥城市云数据中心股份有限公司 | A kind of industrial data multi-dimensional matrix analysis method based on code table mapping configuration technology |
| CN108280099A (en) * | 2017-01-11 | 2018-07-13 | 广州市动景计算机科技有限公司 | Data dictionary management method, apparatus and server |
| CN107273485A (en) * | 2017-06-13 | 2017-10-20 | 苏州弘铭检测科技有限公司 | A kind of data store organisation and database remapping method based on configurable data storehouse |
| CN110572242A (en) * | 2019-09-09 | 2019-12-13 | 四川长虹电器股份有限公司 | Method for collecting data unified codes of injection molding machine |
| CN112162752B (en) * | 2020-09-25 | 2021-05-18 | 成都华数工创科技有限公司 | Internet-based water treatment design method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202058147U (en) * | 2011-05-23 | 2011-11-30 | 北京六所和瑞科技发展有限公司 | Distribution type real-time database management system |
| CN102810091A (en) * | 2011-05-31 | 2012-12-05 | 捷达世软件（深圳）有限公司 | Monitoring data management method and system |
| CN102915383A (en) * | 2011-08-03 | 2013-02-06 | 苏州科技学院 | Regional industrial energy consumption cloud platform and acquisition method of regional industrial energy consumption |
-
2014
- 2014-04-29 CN CN201410175754.0A patent/CN103970914B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202058147U (en) * | 2011-05-23 | 2011-11-30 | 北京六所和瑞科技发展有限公司 | Distribution type real-time database management system |
| CN102810091A (en) * | 2011-05-31 | 2012-12-05 | 捷达世软件（深圳）有限公司 | Monitoring data management method and system |
| CN102915383A (en) * | 2011-08-03 | 2013-02-06 | 苏州科技学院 | Regional industrial energy consumption cloud platform and acquisition method of regional industrial energy consumption |
Non-Patent Citations (2)
| Title |
|---|
| 污水处理厂数据采集系统的设计与应用;陈云翔 等;《能源环境保护》;20110831;第25卷(第4期);第62-64页 * |
| 浅析指挥自动化信息的标准化存储与数据字典;王榕榕 等;《军用标准化》;20030630(第3期);第12-13页摘要，第2-3节，表1 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103970914A (en) | 2014-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103970914B (en) | A kind of acquisition and storage method for isomeric data between sewage treatment plant | |
| Wang et al. | Fast large-scale trajectory clustering | |
| CN110502509B (en) | Traffic big data cleaning method based on Hadoop and Spark framework and related device | |
| CN104317789B (en) | The method for building passenger social network | |
| CN102426609B (en) | Index generation method and index generation device based on MapReduce programming architecture | |
| CN104200369B (en) | Method and device for determining commodity distribution range | |
| CN109033086A (en) | A kind of address resolution, matched method and device | |
| US20140156606A1 (en) | Method and System for Integrating Data Into a Database | |
| JP6300889B2 (en) | System and method for improving extraction performance of atypical text | |
| CN110222039B (en) | Data storage and garbage data cleaning method, device, equipment and storage medium | |
| CN105426375A (en) | Relationship network calculation method and apparatus | |
| CN106021301B (en) | Data comparison system and method for different file formats | |
| WO2016072124A1 (en) | Address/latitude and longitude conversion device and geographic information system in which same is used | |
| CN113536070A (en) | Address resolution method, system, computer equipment and storage medium | |
| CN105893486A (en) | Large-scale graph shortest distance indexing method based on cluster | |
| CN102508866A (en) | Digital addressing-based method for structured storage and rapid processing of command relation tree | |
| CN104731908A (en) | ETL-based data cleaning method | |
| CN106547916A (en) | A kind of user's portrait tag queries method and device | |
| CN110019152A (en) | A kind of big data cleaning method | |
| CN110705297A (en) | Enterprise name-identifying method, system, medium and equipment | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| Dutta et al. | Performance evaluation of south Esk hydrological sensor web: unsupervised machine learning and semantic linked data approach | |
| Ren et al. | Efficient processing of shortest path queries in evolving graph sequences | |
| CN110046343A (en) | Non-standard address conversion is the method that canonical address and canonical address encode | |
| KR102054070B1 (en) | Graph storage management method and graph storage management device for changing tracking and historical graph retrieval |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: High tech Zone Hefei city Anhui province 230088 Magnolia Avenue No. 767 west two big shot Loucq Electromechanical industry building Applicant after: HEFEI CITY CLOUD DATA CENTER CO., LTD. Address before: High tech Zone Hefei city Anhui province 230088 Magnolia Avenue No. 767 west two big shot Loucq Electromechanical industry building Applicant before: City, Hefei company limited of cloud data center |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A method for collecting and storing heterogeneous data between sewage treatment plants Effective date of registration: 20220324 Granted publication date: 20170811 Pledgee: China Construction Bank Corporation Hefei Shushan sub branch Pledgor: HEFEI CITY CLOUD DATA CENTER Co.,Ltd. Registration number: Y2022980003149 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |