CN108776762A - A kind of processing method and processing device of data desensitization - Google Patents

Abstract

This application provides a data desensitization processing method and device, which determine the type of target data; call the corresponding sub-thesaurus in the word segmentation benchmark lexicon according to the type of the target data, and use The corresponding word segmentation method performs word segmentation; according to the type of the target data and the length of the target data, determine the desensitization method of the target data, and use the desensitization method of the target data to segment the target data to obtain Desensitize sensitive data. By segmenting the target data to obtain data with a certain structure, desensitize the part with main sensitive information, mask all or most of the sensitive information, improve the effectiveness of data desensitization, and ensure the security of data assets , to protect the security of customer information to the greatest extent, and avoid customer information leakage caused by abnormal query and export.

Description

A data desensitization processing method and device

technical field

The present invention relates to the technical field of data processing, and more specifically, to a data desensitization processing method and device.

Background technique

In order to implement the national "Network Security Law" on the protection of customer sensitive information, to ensure the security of data assets of power marketing customers, and to protect the legitimate rights and interests of power marketing customers, it is necessary to desensitize the sensitive information of power marketing customers. The purpose is to meet normal business needs. At the same time as necessary, the security of power customer information is protected to the greatest extent, and power customer information leakage caused by abnormal query and export is avoided.

At present, the main rules of power marketing data desensitization mainly adopt the mask desensitization method, retain part of the information, and ensure that the length of the information remains unchanged. The main rules are as follows:

(1) Contact address

Format: The format is not fixed, it is a string of variable length.

Desensitization rules: Reserved in steps according to length, if the length is 5 characters or less, keep the first and last 2 characters; if the length is 6-9 characters, keep the last 5 characters; if the length is 10 characters or more , hide the 4 characters before the last 5 characters; replace the hidden characters with *.

(2) Enterprise account name

Format: The enterprise account name is consistent with the business license, which is the company name and consists of several Chinese characters.

Desensitization rules: Reserved in steps according to the length: if the length is 4 characters or less, keep 1 character at the beginning and the end; if the length is 5-6 characters, keep 2 characters at the beginning and the end; if the length is 7 characters or more odd numbers, hide them The middle 3 characters; the length is 8 characters and above even number, the middle 4 characters are hidden; the hidden characters are replaced by *.

The main disadvantages of the existing power marketing data desensitization rules are:

After desensitization of the two types of power marketing data, electricity address and business households, according to the current data desensitization rules, the non-keywords are masked, but the keywords are still retained. For example, according to the desensitization rules for corporate account names, there may still be sensitive information in the desensitized account name and address, some keywords are reserved, and the desensitization effect is not obvious. As follows: Qingdao Huifeng Motor Manufacturing Co., Ltd. -> Qingdao Huifeng **** Co., Ltd.; Qingdao 2020 Business Service Co., Ltd. -> Qingdao 2****** Service Co., Ltd.

According to the desensitization rules of the contact address, there are similar problems, as follows: 2-1-101, Qilu Ankangyuan Community, Weisan Road, Shanchuan Street, Tianqiao North Residential Committee, Shizhong District, Jinan City, Shandong Province -> Jinan City, Shandong Province 1-101, Qilu Ankangyuan, Weisan Road, Tianqiao North Neighborhood Committee, Shanchuan Street, Central District.

Contents of the invention

In view of this, the present invention discloses a data desensitization processing method and device. Before data desensitization, the target data is segmented by calling a word segmentation benchmark lexicon, so as to realize more effective data desensitization.

In order to realize the foregoing invention object, the specific technical scheme provided by the present invention is as follows:

A processing method for data desensitization, comprising:

Determine the type of target data;

Invoking the corresponding sub-thesaurus in the word segmentation benchmark thesaurus according to the type of the target data, and using a word segmentation method corresponding to the type of the target data to perform word segmentation;

According to the type of the target data and the length of the target data, determine the desensitization method of the target data, and use the desensitization method of the target data to perform desensitization processing on the sensitive data obtained after the word segmentation of the target data .

Optionally, the method also includes:

A word segmentation reference thesaurus is constructed, and the word segmentation reference thesaurus includes a plurality of sub-thesauruses, and each sub-thesaurus includes a type of sensitive word.

Optionally, when the type of the target data is an electrical address, the corresponding sub-thesaurus in the word segmentation benchmark lexicon is invoked according to the type of the target data, and the word segmentation corresponding to the type of the target data is used Methods for word segmentation, including:

The general address sub-thesaurus, the place name sub-thesaurus, the community name sub-thesaurus and the administrative division sub-thesaurus are called, and the target data is segmented by using the maximum positive matching Chinese word segmentation.

Optionally, when the type of the target data is an enterprise account name, the corresponding sub-thesaurus in the word segmentation benchmark lexicon is called according to the type of the target data, and the corresponding sub-thesaurus corresponding to the type of the target data is used. The word segmentation method performs word segmentation, including:

Call the regional collection sub-thesaurus, industry collection sub-thesaurus and company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation.

Optionally, before determining the desensitization method of the target data according to the type of the target data and the length of the target data, the method further includes:

Calculate the accuracy rate of the word segmentation result of the target data;

judging whether the correct rate of the word segmentation result of the target data is greater than a first preset value;

If yes, perform the desensitization method for determining the target data according to the type of the target data and the length of the target data;

If not, segment the target data based on a Hidden Markov Model, and execute the desensitization method for determining the target data according to the type of the target data and the length of the target data.

Optionally, when the type of the target data is an electrical address, according to the type of the target data and the length of the target data, determine a desensitization method for the target data, and use the target data The desensitization method desensitizes the sensitive data obtained after the target data is segmented, including:

judging whether the length of the target data is greater than a second preset value;

When the length of the target data is greater than the second preset value, it is determined that the desensitization method of the target data is the first power address data desensitization method;

Using the first user address data desensitization method, extracting the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data to obtain the remaining part of the data;

Retaining the last 5 digits of the house number data and the data of the provinces, districts, and counties, and masking the remaining data of the target data to obtain desensitized data of the target data;

When the length of the target data is not greater than the second preset value, it is determined that the desensitization method of the target data is the second power address data desensitization method;

Using the second user address data desensitization method, according to the length of the target data, extract the reserved part of the target data according to the first sub-step retention rule, and mask the remaining part of the target data, to obtain the Data after desensitization of the target data.

Optionally, when the type of the target data is an enterprise account name, the desensitization method of the target data is determined according to the type of the target data and the length of the target data, and the target The data desensitization method desensitizes the sensitive data obtained after the target data is segmented, including:

judging whether the length of the target data is greater than a third preset value;

When the length of the target data is greater than the third preset value, it is determined that the desensitization method of the target data is the first enterprise-type account name data desensitization method;

Using the desensitization method of the first enterprise class account name data, extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data, and obtain the remaining data of the font size data and the The rest of the industry data;

Masking the remaining data of the font size data and the remaining data of the industry data, retaining other data of the target data, and obtaining desensitized data of the target data;

When the length of the target data is not greater than the third preset value, it is determined that the desensitization method of the target data is the second enterprise-type account name data desensitization method;

Using the second enterprise-type account name data desensitization method, extracting the retained part of the target data according to the second step-by-step retention rule according to the length of the target data, and masking the remaining part of the target data, Obtain the desensitized data of the target data.

A data desensitization processing device, comprising:

a type determination unit, configured to determine the type of the target data;

The first word segmentation processing unit is used to call the corresponding sub-thesaurus in the word segmentation reference thesaurus according to the type of the target data, and use the word segmentation method corresponding to the type of the target data to perform word segmentation;

The desensitization processing unit is used to determine the desensitization method of the target data according to the type of the target data and the length of the target data, and use the desensitization method of the target data to segment the target data to obtain Desensitize sensitive data.

Optionally, the device also includes:

The thesaurus construction unit is used to construct a word segmentation reference thesaurus, the word segmentation reference thesaurus includes a plurality of sub-thesaurus, and each sub-thesaurus includes a type of sensitive word.

Optionally, when the type of the target data is an electrical address, the first word segmentation processing unit is specifically configured to:

The general address sub-thesaurus, the place name sub-thesaurus, the community name sub-thesaurus and the administrative division sub-thesaurus are called, and the target data is segmented by using the maximum positive matching Chinese word segmentation.

Optionally, when the type of the target data is an enterprise account name, the first word segmentation processing unit is specifically used for:

Call the regional collection sub-thesaurus, industry collection sub-thesaurus and company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation.

Optionally, the device also includes:

A calculation unit, configured to calculate the correct rate of the word segmentation result of the target data;

Judging an end member, used to judge whether the correct rate of the word segmentation result of the target data is greater than a first preset value;

If so, trigger the desensitization processing unit;

If not, a second word segmentation processing unit is triggered, and the second word segmentation processing unit is configured to perform word segmentation on the target data based on a hidden Markov model, and trigger the desensitization processing unit.

Optionally, when the type of the target data is an electrical address, the desensitization processing unit includes:

a first judging subunit, configured to judge whether the length of the target data is greater than a second preset value;

A first determining subunit, configured to determine that the desensitization method of the target data is the first power address data desensitization method when the length of the target data is greater than the second preset value;

The first extraction subunit is used to extract the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data by using the first user address data desensitization method to obtain the remaining data;

The first desensitization processing subunit is used to retain the last 5 digits of the house number data and the province, district and county data, and mask the remaining data of the target data to obtain the desensitization of the target data after the data;

The second determination subunit is configured to determine that the desensitization method of the target data is the second power address data desensitization method when the length of the target data is not greater than the second preset value;

The second desensitization processing subunit is used to adopt the second user address data desensitization method, extract the reserved part of the target data according to the length of the target data according to the first step-by-step retention rule, and process the target The remaining part of the data is masked to obtain the desensitized data of the target data.

Optionally, when the type of the target data is an enterprise account name, the desensitization processing unit includes:

A second judging subunit, configured to judge whether the length of the target data is greater than a third preset value;

A third determination subunit, configured to determine that the desensitization method of the target data is the first desensitization method of enterprise-type account name data when the length of the target data is greater than the third preset value;

The second extraction subunit is used to extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data by using the desensitization method of the first enterprise-type account name data to obtain the The remaining data of the font size data and the remaining data of the industry data;

The third desensitization processing subunit is used to mask the remaining data of the font size data and the remaining data of the industry data, retain other data of the target data, and obtain the desensitized data of the target data;

The fourth determination subunit is used to determine that the desensitization method of the target data is the second desensitization method of enterprise-type account name data when the length of the target data is not greater than the third preset value;

The fourth desensitization processing subunit is used to adopt the second enterprise-type account name data desensitization method, extract the reserved part of the target data according to the second step-by-step retention rule according to the length of the target data, and Mask the remaining part of the target data to obtain desensitized data of the target data.

Compared with the prior art, the beneficial effects of the present invention are as follows:

A data desensitization processing method and device provided by the present invention, before the data desensitization, the target data is segmented by calling the word segmentation benchmark thesaurus to obtain data with a certain structure, and desensitization is performed on the part with main sensitive information , to mask all or most of the sensitive information, improving the effectiveness of data desensitization. According to the type of target data, call the corresponding sub-thesaurus in the benchmark lexicon for word segmentation, and use the word segmentation method corresponding to the type of target data to perform word segmentation, which improves the accuracy of word segmentation, and determines the target data according to the type and length of the target data The desensitization method realizes the differential desensitization of data of different types and lengths, and improves the effectiveness of data desensitization.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a flowchart of a data desensitization processing method disclosed in an embodiment of the present invention;

FIG. 2 is a schematic diagram of a general address sub-thesaurus disclosed in an embodiment of the present invention;

Fig. 3 is a schematic diagram of the sub-thesaurus of the place name database disclosed in the embodiment of the present invention;

FIG. 4 is a schematic diagram of a cell name sub-thesaurus disclosed in an embodiment of the present invention;

Fig. 5 is the schematic diagram of the set sub-thesaurus of administrative division division disclosed by the embodiment of the present invention;

Fig. 6 is a schematic diagram of the regional set sub-thesaurus disclosed in the embodiment of the present invention;

Fig. 7 is a schematic diagram of an industry collection sub-thesaurus disclosed in an embodiment of the present invention;

Fig. 8 is a schematic diagram of the company organization collection sub-thesaurus disclosed in the embodiment of the present invention;

Fig. 9 is a schematic diagram of the maximum forward matching Chinese word segmentation method disclosed in the embodiment of the present invention;

Fig. 10 is a flow chart of a desensitization processing method for electricity address data disclosed in an embodiment of the present invention;

Fig. 11 is a flowchart of a desensitization processing method for enterprise account name data disclosed in an embodiment of the present invention;

Fig. 12 is a flowchart of another data desensitization processing method disclosed in the embodiment of the present invention;

Fig. 13 is a schematic structural diagram of a data desensitization processing device disclosed in an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to Figure 1. This embodiment discloses a data desensitization processing method, which specifically includes the following steps:

S101: Determine the type of target data;

The target data is the data that needs to be desensitized. The type of target data may include phone data, address data, user name data, bank account data, etc.

S102: Call the corresponding sub-thesaurus in the word segmentation reference thesaurus according to the type of the target data, and use the word segmentation method corresponding to the type of the target data to perform word segmentation;

Word segmentation is to divide a sequence of Chinese characters into individual words. Word segmentation is the process of recombining continuous word sequences into word sequences according to certain specifications.

In order to more accurately segment the target data, according to the type of the target data, the corresponding sub-thesaurus in the benchmark lexicon for word segmentation is called to segment the target data.

It should be noted that the data desensitization processing method also includes:

Build a word segmentation benchmark thesaurus.

The word segmentation benchmark lexicon includes multiple sub-thesauruses, and each sub-thesaurus includes a type of sensitive words.

Please refer to Figures 2 to 8, which are the general address sub-thesaurus, place name sub-thesaurus, community name sub-thesaurus, administrative division collection sub-thesaurus, regional collection sub-thesaurus, and industry collection sub-thesaurus in the word segmentation benchmark lexicon. and company organization collection sub-thesaurus.

In order to more accurately segment the target data, the corresponding sub-thesaurus in the word segmentation reference thesaurus is invoked according to the type of the target data, and the word segmentation method corresponding to the type of the target data is used for word segmentation. For example, when the type of the target data is an electricity address, call the general address sub-thesaurus, place name sub-thesaurus, community name sub-thesaurus and administrative division set sub-thesaurus, and use the maximum positive matching Chinese word segmentation to pair the The target data is word-segmented. When the type of the target data is an enterprise account name, call the regional collection sub-thesaurus, the industry collection sub-thesaurus and the company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation.

As shown in Figure 9, the maximum positive matching Chinese word segmentation algorithm is used in the word segmentation of the electricity address data, and the specific algorithm is as follows:

Match several consecutive characters in the target data with the vocabulary from left to right, and if they match, a word is segmented. But there is a problem here: to achieve the maximum match, it is not possible to split the first match. Such as text to be segmented:

content[]＝{"Hong", "Mountain", "Street", "Tao", "Shuang", "River", "Community", "District",...}

Vocabulary: dict[]＝{"Changsha City", "Kaifu District", "Hongshan", "Hongshan Street",...}

(1) Starting from content[1], when content[2] is scanned, it is found that "Hongshan" is already in the vocabulary dict[]. But it can't be segmented yet, because we don't know whether the following words can form longer words (maximum match);

(2) Continue to scan content[3] and find that "Hongshan Street" is not a word in dict[]. But we are still not sure whether the "Hongshan" found earlier is already the largest word, because "Hongshan Street" is the prefix of dict[2];

(3) Scan content[4] and find that "Hongshan Street" is a word in dict[]. Continue to scan;

(4) When scanning content[5], it is found that "Hongshan Street Double" is not a word in the vocabulary, nor is it a prefix of a word. Therefore, the biggest word in the front can be cut out - "Hongshan street".

It can be seen that the maximum matched word must ensure that the next scan is not a word or a prefix of a word in the vocabulary before it can end. Use the maximum forward matching algorithm to continue the loop to complete the remaining word segmentation. For example, "Unit 1706, Unit 2, Building 10, Phase III, Contemporary Wanguocheng, No. 199, Fuyuan West Road, Shuanghe Community, Hongshan Street, Kaifu District, Changsha City" is as follows:

"Changsha | Kaifu District | Hongshan Street | Shuanghe Community | Fuyuan West Road | No. 199 | Contemporary Wanguocheng Phase III | 10 | Building | Second | Unit | 1706".

The two-way maximum matching Chinese word segmentation method is used in the word segmentation of enterprise account name data. The two-way maximum matching Chinese word segmentation method first performs the maximum forward matching and maximum reverse matching Chinese word segmentation respectively, and compares the word segmentation results on this basis, and adopts different word segmentation strategies according to different results. Good, the principle that the fewer non-dictionary words and single-character words is better, select one of the word segmentation results to output.

The maximum forward matching Chinese word segmentation algorithm has been described in detail. The maximum reverse matching Chinese word segmentation algorithm is similar to the maximum forward matching algorithm, the difference is the scanning direction, which takes substrings from right to left for matching. The algorithm flow can be described as:

(1) Input the sentence content to be segmented after preprocessing, and initialize index=content.length;

(2) obtain the length of each sub-dictionary in the dictionary database;

(3) Obtain the length of the participle word and compare it with the longest sub-dictionary in the dictionary database. If the maximum length of the sub-dictionary is greater than the length of the word to be segmented, then take the string remaining in the word to be segmented as the maximum length, otherwise use the maximum length participle;

(4) search for the same sub-dictionary as the current maximum matching length by dichotomy, if find the dictionary then turn (5), otherwise the maximum length minus one turn (4);

(5) Obtain the string SubStr to be segmented, look for the string in the dictionary, if found, add the string to the List, if not found, determine whether SubStr is greater than 1, if greater than 1, delete the last SubStr Word turns (5), otherwise puts segmentation mark, turns (6);

(6) Determine whether the Index is greater than 1, if it is less, go to (3) otherwise save the List and exit.

The two-way maximum matching algorithm combines the forward matching and reverse matching algorithms. For the character string to be divided, firstly use the maximum forward matching and maximum reverse matching algorithms for word segmentation, and compare the results of word segmentation, comparing the forward and reverse two A maximum match, returns the word segmentation result; when the word segmentation results in the two directions are consistent, the returned strings are inconsistent, and return the smaller length; when the length is the same, return the reverse. The steps of the two-way maximum matching Chinese word segmentation algorithm are as follows:

(1) Input the content of the sentence to be segmented;

(2) After preprocessing the content, use the maximum forward matching algorithm and the maximum reverse matching algorithm to perform word segmentation respectively, compare the word segmentation results, if the word segmentation results are exactly the same, go to (3), if the word segmentation results are different, go to (4) ;

(3) arbitrarily select a word segmentation result, and output the word segmentation result to the end of the algorithm;

(4) Compare whether the number of word segmentation is the same, if the same, select the reverse word segmentation result, output the word segmentation result, and the algorithm ends; otherwise select the word segmentation result with a smaller number of word segmentation to output, and the algorithm ends.

S103: Determine the desensitization method of the target data according to the type of the target data and the length of the target data, and use the desensitization method of the target data to desensitize the sensitive data obtained after word segmentation of the target data Sensitive handling.

Please refer to Fig. 10, when the type of the target data is a power consumption address, the execution process of S103 is as follows:

S201: Determine whether the length of the target data is greater than a second preset value; if yes, execute S202, if not, execute S203:

S202: Determine that the desensitization method of the target data is the first power usage address data desensitization method;

S204: Using the first user address data desensitization method, extract the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data to obtain the remaining data;

S205: Reserving the last 5 digits of the house number data and the data of the province, district and county, and masking the remaining data of the target data to obtain desensitized data of the target data;

S203: Determine that the desensitization method of the target data is the second power usage address data desensitization method;

S206: Using the second user address data desensitization method, extracting the reserved part of the target data according to the length of the target data according to the first hierarchical retention rule, and masking the remaining part of the target data, Obtain the desensitized data of the target data.

For example, for electricity address data with a length of 10 words or less, data desensitization is carried out according to the second user address data desensitization method, and the data is reserved in steps according to the length. If the length is 5 words or less, the first word and the last 2 characters; if the length is 6-9 characters, keep the last 5 characters.

For electricity address data with a length of 10 words or more, perform data desensitization according to the first user address data desensitization method. Electricity address is generally composed of province, city, district/county, street/township committee/village, road, community, and house number. The last 5 digits of the house number are reserved, provinces, cities, districts and counties are reserved, and all other parts are replaced by *. As follows:

2-1-101, Qilu Ankangyuan Community, Weisan Road, North Neighborhood Committee, Weisan Road, Shanchuan Street, Shizhong District, Jinan City, Shandong Province -> Shizhong District, Jinan City, Shandong Province****************** *****1-101.

Please refer to Fig. 11, when the type of the target data is a power consumption address, the execution process of S103 is as follows:

S301: Determine whether the length of the target data is greater than a third preset value; if yes, perform S302, and if not, perform S303;

S302: Determine that the desensitization method of the target data is the desensitization method of the first enterprise-type account name data;

S304: Using the desensitization method for the first enterprise-type account name data, extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data, and obtain the remaining data of the font size data and the remainder of said industry data;

S305: Mask the remaining data of the font size data and the remaining data of the industry data, retain other data of the target data, and obtain desensitized data of the target data;

S303: Determine that the desensitization method of the target data is the second enterprise-type account name data desensitization method;

S306: Using the second desensitization method for enterprise-type account name data, extract the reserved part of the target data according to the length of the target data according to the second ladder retention rule, and mask the remaining part of the target data code to obtain the desensitized data of the target data.

For example, for the enterprise account name data with a length of less than 6 characters, data desensitization is performed according to the second electricity address data desensitization method, and the length is divided into steps. For the length of 4 characters and less, the first and last characters are reserved; If the length is 5-6 characters, keep 2 characters at the beginning and end.

For enterprise account name data with a length of 6 characters or more, the data is desensitized according to the data desensitization method of the first power consumption address. An enterprise account name generally consists of four parts: region, trade name, industry, and company organization. The regions and organizations before and after are kept unchanged, and the font size and industry are masked. For the part of font size, the first character is reserved, and all other parts are replaced with *; for the industry part, the last character is reserved, and all other parts are replaced with *. As follows:

Qingdao Huifeng Motor Manufacturing Co., Ltd.->Qingdao Hui**** Manufacturing Co., Ltd.;

Qingdao 2020 Business Service Co., Ltd. -> Qingdao 2020 Service Co., Ltd.

In the data desensitization processing method disclosed in this embodiment, before the data desensitization, the target data is segmented by calling the word segmentation benchmark lexicon to obtain data with a certain structure, and the part with main sensitive information is desensitized. Masking all or most of the sensitive information improves the effectiveness of data desensitization. According to the type of target data, call the corresponding sub-thesaurus in the benchmark lexicon for word segmentation, and use the word segmentation method corresponding to the type of target data to perform word segmentation, which improves the accuracy of word segmentation, and determines the target data according to the type and length of the target data The desensitization method realizes the differential desensitization of data of different types and lengths, and improves the effectiveness of data desensitization.

Please refer to Figure 12, this embodiment discloses another data desensitization processing method, which specifically includes the following steps:

S401: Determine the type of target data;

S402: Call the corresponding sub-thesaurus in the word segmentation reference thesaurus according to the type of the target data, and use the word segmentation method corresponding to the type of the target data to perform word segmentation;

S403: Calculate the accuracy rate of the word segmentation result of the target data;

S404: Determine whether the correct rate of the word segmentation result of the target data is greater than the first preset value; if yes, execute S405; if not, execute S406;

S405: Determine the desensitization method of the target data according to the type of the target data and the length of the target data, and use the desensitization method of the target data to desensitize the sensitive data obtained after word segmentation of the target data sensitive treatment;

S406: Segment the target data based on the Hidden Markov Model, and perform S405.

The HMM Hidden Markov Model is used to perform Chinese word segmentation processing on the two types of data of enterprise account name and electricity address. The HMM algorithm can obtain a higher segmentation accuracy when the training corpus is large enough and covers enough fields. This type of word segmentation algorithm is based on the artificially marked part-of-speech and statistical features to model Chinese, that is, to estimate the model parameters based on the observed data (marked corpus), that is, to train. In the word segmentation stage, the probability of various word segmentations is calculated through the model, and the word segmentation result with the highest probability is taken as the final result. The common sequence labeling model has the HMM algorithm, which can handle ambiguity and unregistered words well, and the effect is better than that based on string matching.

Hidden Markov model is a double stochastic process, we do not know the specific state sequence, only the probability of state transition, that is, the state transition process of the model is unobservable (concealed), while the random process of observable events is Stochastic functions for covert state transition processes.

The composition of HMM includes:

The number of states in the model is N;

The number M of different symbols that may be output from each state;

State transition probability matrix A=a _ij , where a _ij is the probability of transitioning from state S _i to state S _j ;

The probability distribution matrix of observing a specific symbol O _k from the state C _j is: B=b _j (k), the probability of observing the symbol is also called the symbol emission probability;

The probability distribution of the initial state is: π={π _i }.

Generally, an HMM is recorded as a five-tuple μ=(C, K, A, B, π), where C is the set of states, O is the set of output symbols, and π, A, and B are the initial state Probability distributions, state transition probabilities, and symbol emission probabilities.

Chinese word segmentation uses corpus to train HMM. Using the classic character annotation model, the set C of four types of labels is C={B, E, M, S}, and its meaning is as follows:

B: the beginning of a word

E: end of a word

M: middle of a word

S: a single character into a word

After marking with four types of labels, you can start to build an HMM model using statistical methods. The label classification of each character is only affected by the previous character classification. The state transition matrix A and the symbol emission probability B of the HMM are obtained. in:

In the formula, C={B, E, M, S}, O={character set}, and Count represents the frequency. When calculating B _ij , due to the sparsity of the data, many characters do not appear in the training set, which leads to a result with a probability of 0 appearing in B. In order to remedy this problem, a data smoothing technique of adding 1 is used, namely:

We set the initial vector π={0.5, 0.0, 0.0, 0.5}, M and E cannot appear at the beginning of the sentence. So far, the HMM model has been constructed. Based on this HMM model, for an observation sequence, a hidden sequence {B, E, M, S} is obtained with the Viterbi algorithm.

The Viterbi search algorithm is:

1. Initialization: δ ₁ (i) = π _i b _i (O1), 1≤i≤N,

The most probable path variable:

2. Recursive calculation:

3. Memory rollback path:

4. End:

Get the path (state sequence) by backtracking:

The time complexity of the Viterbi algorithm is O(N ² T). For example, "Unit 1706, Unit 2, Building 10, Phase III, Contemporary Wanguocheng, No. 199, Fuyuan West Road, Shuanghe Community, Hongshan Street, Kaifu District, Changsha City" is the output status sequence of the address:

"BMEBMEBMMEBMMEBMMEBMMEBMMMEBMMMMMEBMEBMEBMME"

According to this state sequence, Chinese word segmentation can be performed as follows:

"BME|BME|BMME|BMME|BMME|BMME|BMMMMME|BME|BME|BMME"

The final Chinese word segmentation results are as follows:

"Changsha City|Kaifu District|Hongshan Street|Shuanghe Community|Fuyuan West Road|No. 199|Phase III of Contemporary World City|Building 10|Unit 2|1706".

The data desensitization processing method disclosed in this embodiment first uses the maximum forward matching method or the two-way maximum matching Chinese word segmentation method with less algorithm complexity to perform word segmentation processing on the target word, which ensures the processing speed of word segmentation processing. Calculate the correct rate of the word segmentation result. When the correct rate of the word segmentation result is lower than the threshold, the hidden Markov model with high algorithm complexity but high word segmentation accuracy is used to segment the target data to ensure the accuracy of the word segmentation result. .

Based on a data desensitization processing method disclosed in the above embodiment, please refer to FIG. 13. This embodiment discloses a data desensitization processing device correspondingly, including:

A type determining unit 501, configured to determine the type of the target data;

The first word segmentation processing unit 502 is used to call the corresponding sub-thesaurus in the word segmentation benchmark lexicon according to the type of the target data, and use the word segmentation method corresponding to the type of the target data to perform word segmentation;

A desensitization processing unit 503, configured to determine a desensitization method for the target data according to the type of the target data and the length of the target data, and use the desensitization method for the target data to segment the target data The obtained sensitive data is desensitized.

Optionally, the device also includes:

The thesaurus construction unit is used to construct a word segmentation reference thesaurus, the word segmentation reference thesaurus includes a plurality of sub-thesaurus, and each sub-thesaurus includes a type of sensitive word.

Optionally, when the type of the target data is an electrical address, the first word segmentation processing unit 502 is specifically configured to:

The general address sub-thesaurus, the place name sub-thesaurus, the community name sub-thesaurus and the administrative division sub-thesaurus are called, and the target data is segmented by using the maximum positive matching Chinese word segmentation.

Optionally, when the type of the target data is an enterprise account name, the first word segmentation processing unit 502 is specifically configured to:

Call the regional collection sub-thesaurus, industry collection sub-thesaurus and company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation.

Optionally, the device also includes:

A calculation unit, configured to calculate the correct rate of the word segmentation result of the target data;

Judging an end member, used to judge whether the correct rate of the word segmentation result of the target data is greater than a first preset value;

If so, trigger the desensitization processing unit;

If not, a second word segmentation processing unit is triggered, and the second word segmentation processing unit is configured to perform word segmentation on the target data based on a hidden Markov model, and trigger the desensitization processing unit.

Optionally, when the type of the target data is an electrical address, the desensitization processing unit 503 includes:

a first judging subunit, configured to judge whether the length of the target data is greater than a second preset value;

A first determining subunit, configured to determine that the desensitization method of the target data is the first power address data desensitization method when the length of the target data is greater than the second preset value;

The first extraction subunit is used to extract the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data by using the first user address data desensitization method to obtain the remaining data;

The first desensitization processing subunit is used to retain the last 5 digits of the house number data and the province, district and county data, and mask the remaining data of the target data to obtain the desensitization of the target data after the data;

The second determination subunit is configured to determine that the desensitization method of the target data is the second power address data desensitization method when the length of the target data is not greater than the second preset value;

The second desensitization processing subunit is used to adopt the second user address data desensitization method, extract the reserved part of the target data according to the length of the target data according to the first step-by-step retention rule, and process the target The remaining part of the data is masked to obtain the desensitized data of the target data.

Optionally, when the type of the target data is an enterprise account name, the desensitization processing unit 503 includes:

A second judging subunit, configured to judge whether the length of the target data is greater than a third preset value;

A third determination subunit, configured to determine that the desensitization method of the target data is the first desensitization method of enterprise-type account name data when the length of the target data is greater than the third preset value;

The second extraction subunit is used to extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data by using the desensitization method of the first enterprise-type account name data to obtain the The remaining data of the font size data and the remaining data of the industry data;

The third desensitization processing subunit is used to mask the remaining data of the font size data and the remaining data of the industry data, retain other data of the target data, and obtain the desensitized data of the target data;

The fourth determination subunit is used to determine that the desensitization method of the target data is the second desensitization method of enterprise-type account name data when the length of the target data is not greater than the third preset value;

The fourth desensitization processing subunit is used to adopt the second enterprise-type account name data desensitization method, extract the reserved part of the target data according to the second step-by-step retention rule according to the length of the target data, and Mask the remaining part of the target data to obtain desensitized data of the target data.

In the data desensitization processing device disclosed in this embodiment, before the data desensitization, the target data is segmented by calling the word segmentation benchmark thesaurus to obtain data with a certain structure, and desensitization is performed on the part with main sensitive information. Masking all or most of the sensitive information improves the effectiveness of data desensitization. According to the type of target data, call the corresponding sub-thesaurus in the benchmark lexicon for word segmentation, and use the word segmentation method corresponding to the type of target data to perform word segmentation, which improves the accuracy of word segmentation, and determines the target data according to the type and length of the target data The desensitization method realizes the differential desensitization of data of different types and lengths, and improves the effectiveness of data desensitization.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (14)

1. A processing method for data desensitization, comprising: Determine the type of target data; Invoking the corresponding sub-thesaurus in the word segmentation benchmark thesaurus according to the type of the target data, and using a word segmentation method corresponding to the type of the target data to perform word segmentation; According to the type of the target data and the length of the target data, determine the desensitization method of the target data, and use the desensitization method of the target data to perform desensitization processing on the sensitive data obtained after the word segmentation of the target data . 2. The method according to claim 1, characterized in that the method further comprises: A word segmentation reference thesaurus is constructed, and the word segmentation reference thesaurus includes a plurality of sub-thesauruses, and each sub-thesaurus includes a type of sensitive word. 3. method according to claim 1, is characterized in that, when the type of described target data is electricity address, described according to the type of described target data calls corresponding sub-thesaurus in word segmentation benchmark lexicon, adopts The word segmentation method corresponding to the type of the target data performs word segmentation, including: The general address sub-thesaurus, the place name sub-thesaurus, the community name sub-thesaurus and the administrative division sub-thesaurus are called, and the target data is segmented by using the maximum positive matching Chinese word segmentation. 4. method according to claim 1, it is characterized in that, when the type of described target data is enterprise class account name, described according to the type of described target data calls the corresponding sub-thesaurus in the participle benchmark lexicon, A word segmentation method corresponding to the type of the target data is used for word segmentation, including: Call the regional collection sub-thesaurus, industry collection sub-thesaurus and company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation. 5. The method according to claim 1, wherein, before determining the desensitization method of the target data according to the type of the target data and the length of the target data, the method further comprises: Calculate the accuracy rate of the word segmentation result of the target data; judging whether the correct rate of the word segmentation result of the target data is greater than a first preset value; If yes, perform the desensitization method for determining the target data according to the type of the target data and the length of the target data; If not, segment the target data based on a Hidden Markov Model, and execute the desensitization method for determining the target data according to the type of the target data and the length of the target data. 6. The method according to claim 1, wherein when the type of the target data is an electrical address, the target data is determined according to the type of the target data and the length of the target data desensitization method of the target data, and use the desensitization method of the target data to desensitize the sensitive data obtained after the target data is segmented, including: judging whether the length of the target data is greater than a second preset value; When the length of the target data is greater than the second preset value, it is determined that the desensitization method of the target data is the first power address data desensitization method; Using the first user address data desensitization method, extracting the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data to obtain the remaining part of the data; Retaining the last 5 digits of the house number data and the data of the provinces, districts, and counties, and masking the remaining data of the target data to obtain desensitized data of the target data; When the length of the target data is not greater than the second preset value, it is determined that the desensitization method of the target data is the second power address data desensitization method; Using the second user address data desensitization method, according to the length of the target data, extract the reserved part of the target data according to the first sub-step retention rule, and mask the remaining part of the target data, to obtain the Data after desensitization of the target data. 7. The method according to claim 1, wherein, when the type of the target data is an enterprise account name, the target is determined according to the type of the target data and the length of the target data Data desensitization method, and using the target data desensitization method to desensitize the sensitive data obtained after the target data is segmented, including: judging whether the length of the target data is greater than a third preset value; When the length of the target data is greater than the third preset value, it is determined that the desensitization method of the target data is the first enterprise-type account name data desensitization method; Using the desensitization method of the first enterprise class account name data, extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data, and obtain the remaining data of the font size data and the The rest of the industry data; Masking the remaining data of the font size data and the remaining data of the industry data, retaining other data of the target data, and obtaining desensitized data of the target data; When the length of the target data is not greater than the third preset value, it is determined that the desensitization method of the target data is the second enterprise-type account name data desensitization method; Using the second enterprise-type account name data desensitization method, extracting the retained part of the target data according to the second step-by-step retention rule according to the length of the target data, and masking the remaining part of the target data, Obtain the desensitized data of the target data. 8. A processing device for data desensitization, comprising: a type determination unit, configured to determine the type of the target data; The first word segmentation processing unit is used to call the corresponding sub-thesaurus in the word segmentation reference thesaurus according to the type of the target data, and use the word segmentation method corresponding to the type of the target data to perform word segmentation; The desensitization processing unit is used to determine the desensitization method of the target data according to the type of the target data and the length of the target data, and use the desensitization method of the target data to segment the target data to obtain Desensitize sensitive data. 9. The device according to claim 8, further comprising: The thesaurus construction unit is used to construct a word segmentation reference thesaurus, the word segmentation reference thesaurus includes a plurality of sub-thesaurus, and each sub-thesaurus includes a type of sensitive word. 10. The device according to claim 8, wherein when the type of the target data is an electrical address, the first word segmentation processing unit is specifically used for: The general address sub-thesaurus, the place name sub-thesaurus, the community name sub-thesaurus and the administrative division sub-thesaurus are called, and the target data is segmented by using the maximum positive matching Chinese word segmentation. 11. The device according to claim 8, wherein when the type of the target data is an enterprise account name, the first word segmentation processing unit is specifically used for: Call the regional collection sub-thesaurus, industry collection sub-thesaurus and company organization collection sub-thesaurus, and use the two-way maximum matching Chinese word segmentation method for word segmentation. 12. The device according to claim 8, further comprising: A calculation unit, configured to calculate the correct rate of the word segmentation result of the target data; Judging an end member, used to judge whether the correct rate of the word segmentation result of the target data is greater than a first preset value; If so, trigger the desensitization processing unit; If not, a second word segmentation processing unit is triggered, and the second word segmentation processing unit is configured to perform word segmentation on the target data based on a hidden Markov model, and trigger the desensitization processing unit. 13. The device according to claim 8, wherein when the type of the target data is an electrical address, the desensitization processing unit comprises: a first judging subunit, configured to judge whether the length of the target data is greater than a second preset value; A first determining subunit, configured to determine that the desensitization method of the target data is the first power address data desensitization method when the length of the target data is greater than the second preset value; The first extraction subunit is used to extract the last 5 digits of the house number data and the province, district and county data from the word segmentation result of the target data by using the first user address data desensitization method to obtain the remaining data; The first desensitization processing subunit is used to retain the last 5 digits of the house number data and the province, district and county data, and mask the remaining data of the target data to obtain the desensitization of the target data after the data; The second determination subunit is configured to determine that the desensitization method of the target data is the second power address data desensitization method when the length of the target data is not greater than the second preset value; The second desensitization processing subunit is used to adopt the second user address data desensitization method, extract the reserved part of the target data according to the length of the target data according to the first step-by-step retention rule, and process the target The remaining part of the data is masked to obtain the desensitized data of the target data. 14. The method according to claim 8, wherein when the type of the target data is an enterprise account name, the desensitization processing unit includes: A second judging subunit, configured to judge whether the length of the target data is greater than a third preset value; A third determination subunit, configured to determine that the desensitization method of the target data is the first desensitization method of enterprise-type account name data when the length of the target data is greater than the third preset value; The second extraction subunit is used to extract the first character of the font size data and the last character of the industry data from the word segmentation result of the target data by using the desensitization method of the first enterprise-type account name data to obtain the The remaining data of the font size data and the remaining data of the industry data; The third desensitization processing subunit is used to mask the remaining data of the font size data and the remaining data of the industry data, retain other data of the target data, and obtain the desensitized data of the target data; The fourth determination subunit is used to determine that the desensitization method of the target data is the second desensitization method of enterprise-type account name data when the length of the target data is not greater than the third preset value; The fourth desensitization processing subunit is used to adopt the second enterprise-type account name data desensitization method, extract the reserved part of the target data according to the second step-by-step retention rule according to the length of the target data, and Mask the remaining part of the target data to obtain desensitized data of the target data.