CN117668811A - Offline feature management platform, offline feature management method, electronic equipment and storage medium - Google Patents
Offline feature management platform, offline feature management method, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117668811A CN117668811A CN202311685890.XA CN202311685890A CN117668811A CN 117668811 A CN117668811 A CN 117668811A CN 202311685890 A CN202311685890 A CN 202311685890A CN 117668811 A CN117668811 A CN 117668811A
- Authority
- CN
- China
- Prior art keywords
- feature
- target
- target feature
- request
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 91
- 238000003860 storage Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 claims description 27
- 238000012795 verification Methods 0.000 claims description 24
- 238000012550 audit Methods 0.000 claims description 13
- 230000006870 function Effects 0.000 description 12
- 238000004590 computer program Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 230000004048 modification Effects 0.000 description 9
- 238000012986 modification Methods 0.000 description 9
- 230000008859 change Effects 0.000 description 8
- 238000005192 partition Methods 0.000 description 8
- 238000012217 deletion Methods 0.000 description 6
- 230000037430 deletion Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an offline feature management platform, an offline feature management method, an offline feature management electronic device and a offline feature management storage medium. Because the user authority is verified before the database is requested according to the target characteristics sent by the user, irrelevant personnel are prevented from modifying the characteristic library, and the safety of the characteristic library is ensured. In addition, by carrying out distributed storage on each feature, namely storing each feature through a plurality of machines, a large number of feature anomalies caused by single machine faults are avoided, and the security of the feature library is further improved.
Description
Technical Field
The present invention relates to the field of big data, and in particular, to an offline feature management platform, an offline feature management method, an electronic device, and a storage medium.
Background
The advent of the big data age has also driven the development of various fields such as science and technology, finance, manufacturing, etc. for the efficient use of data. Artificial intelligence is also rapidly developing as a superstructure for big data. The artificial intelligence utilizes massive data to generate a predictive model to aid business decisions. And as one of the key underlying cores of artificial intelligence, management and storage of data is critical.
Enterprises that conduct artificial intelligence development typically have databases, i.e., feature libraries, for uniformly storing features. The more abundant the features in the feature library and the larger the feature quantity, the more complex and more powerful models can be trained by enterprises. Thus, enterprises are continually accumulating and managing rich features.
An enterprise typically designs an offline feature management scheme to manage a feature library, where the offline feature management scheme refers to a management scheme that performs feature extraction, conversion, and storage on historical data without network or cloud computing.
Disclosure of Invention
In view of this, the embodiments of the present invention provide an offline feature management platform, a method, an electronic device, and a storage medium, so as to improve the security of offline features.
In a first aspect of the present invention, there is provided an offline feature management platform, the platform comprising: the system comprises a verification module, a management module and a feature library;
the feature library is used for storing all the features in a distributed manner;
the verification module is used for determining whether the target user has the management authority of the target feature included in the target feature request or not under the condition that the target feature request sent by the target user is received; sending the target feature request to a management module under the condition that the target user has the management authority;
the management module is used for changing the feature library according to the target feature request; and updating the configuration file of the feature library according to the changed feature library.
In a possible embodiment, the determining whether the target user has the management authority of the target feature included in the target feature request includes:
and determining whether the target user belongs to a target user group corresponding to the target feature included in the target feature request, and determining that the target user has the management authority of the target feature under the condition that the target user belongs to the target user group.
In a possible embodiment, the verification module is further configured to determine the target feature request type; transmitting the target feature request to the management module under the condition that the target feature request type is a preset white list type; transmitting the target feature request to a preset client under the condition that the target feature request type is not the preset white list type; and under the condition that the audit passing information returned by the preset client is received, sending the target feature request to the management module.
In a possible embodiment, the verification module is further configured to determine whether a feature update time accords with a preset update period, and output alarm information when the feature update time does not accord with the preset update period; and under the condition that the characteristic updating time accords with the preset updating period, sending the target characteristic request to the management module.
In one possible embodiment, the platform further comprises:
a metadata base for storing description information of each of the features, the description information including a storage location;
before the feature library is changed according to the target feature request, the method further comprises the following steps: adding the target feature request to the metadata repository,
and the management module is also used for updating the metadata base according to the changed feature base.
In another aspect of the present invention, there is provided an offline feature management method applied to the offline feature management platform of claim 1; the method comprises the following steps:
under the condition that a target feature request sent by a target user is received, determining whether the target user has the management authority of a target feature included in the target feature request;
under the condition that the target user has the management authority, changing a feature library according to the target feature request; the feature library is used for storing features in a distributed mode;
and updating the configuration file of the feature library according to the changed feature library.
In a possible embodiment, the determining whether the target user has the management authority of the target feature included in the target feature request includes:
and determining whether the target user belongs to a target user group corresponding to the target feature included in the target feature request, and determining that the target user has the management authority of the target feature under the condition that the target user belongs to the target user group.
In one possible embodiment, the method further comprises:
determining the target feature request type;
transmitting the target feature request to the management module under the condition that the target feature request type is a preset white list type;
transmitting the target feature request to a preset client under the condition that the target feature request type is not the preset white list type;
and under the condition that the audit passing information returned by the preset client is received, sending the target feature request to the management module.
In another aspect of the present invention, there is provided an electronic apparatus including:
a processor; and
a memory in which a program is stored,
wherein the program comprises instructions which, when executed by the processor, cause the processor to perform any of the offline feature management methods described above.
In another aspect of the invention, a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform any of the offline feature management methods described above is provided.
According to one or more technical schemes provided by the embodiment of the invention, the verification module is used for verifying the target user sending the target feature request, and after the management authority of the target user with the target feature is determined, the management module is used for modifying the feature library according to the target feature request and modifying the corresponding configuration file. Because the user authority is verified before the database is requested according to the target characteristics sent by the user, irrelevant personnel are prevented from modifying the characteristic library, and the safety of the characteristic library is ensured. In addition, by carrying out distributed storage on each feature, namely storing each feature through a plurality of machines, a large number of feature anomalies caused by single machine faults are avoided, and the security of the feature library is further improved.
Drawings
Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments with reference to the following drawings, in which:
FIG. 1 is a schematic structural diagram of an offline feature management platform according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of authority setting for different request types according to an embodiment of the present invention;
FIG. 3 is a flow chart of a cross-user group access feature in an embodiment of the invention;
FIG. 4 is a schematic flow chart of auditing user rights in an embodiment of the present invention;
FIG. 5 is a flow chart of verifying different types of requests according to an embodiment of the present invention;
FIG. 6 is a schematic flow chart of feature binning in an embodiment of the present invention;
FIG. 7 is a schematic diagram illustrating an implementation procedure of the offline feature management platform according to an embodiment of the present invention;
FIG. 8 is a schematic flow chart of an offline feature management method according to an embodiment of the present invention;
fig. 9 shows a block diagram of an exemplary electronic device that can be used to implement an embodiment of the invention.
Detailed Description
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While the invention is susceptible of embodiment in the drawings, it is to be understood that the invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided to provide a more thorough and complete understanding of the invention. It should be understood that the drawings and embodiments of the invention are for illustration purposes only and are not intended to limit the scope of the present invention.
It should be understood that the various steps recited in the method embodiments of the present invention may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the invention is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below. It should be noted that the terms "first," "second," and the like herein are merely used for distinguishing between different devices, modules, or units and not for limiting the order or interdependence of the functions performed by such devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those skilled in the art will appreciate that "one or more" is intended to be construed as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of such messages or information.
In order to improve the safety of offline features, the invention provides an offline feature management platform, an offline feature management method, electronic equipment and a storage medium. The offline feature management platform provided by the invention can be applied to any electronic equipment with an offline feature management function, wherein the electronic equipment can be a server, a computer, a mobile terminal and the like. The following describes the aspects of the invention with reference to the accompanying drawings:
fig. 1 is a schematic architecture diagram of an offline feature management platform according to an embodiment of the present invention, where the offline feature management platform 100 may include: a verification module 101, a management module 102, and a feature library 103;
the feature library 103 is used for storing various features in a distributed manner;
the verification module 101 is configured to determine, when a target feature request sent by a target user is received, whether the target user has a management authority of a target feature included in the target feature request; in the case that the target user has the management authority, sending the target feature request to a management module 102;
the management module 102 is configured to change the feature library 103 according to the target feature request; and updating the configuration file of the feature library 103 according to the changed feature library 103.
By applying the embodiment of the invention, the verification module is used for verifying the target user sending the target feature request, and after the management authority of the target user with the target feature is determined, the management module is used for modifying the feature library according to the target feature request and modifying the corresponding configuration file. Because the user authority is verified before the database is requested according to the target characteristics sent by the user, irrelevant personnel are prevented from modifying the characteristic library, and the safety of the characteristic library is ensured. In addition, by carrying out distributed storage on each feature, namely storing each feature through a plurality of machines, a large number of feature anomalies caused by single machine faults are avoided, and the security of the feature library is further improved.
The feature library stores the features by adopting a distributed storage mode. The disk space on a plurality of machines is used through a network, the scattered storage spaces form a virtual storage device, the characteristics are stored in each machine of an enterprise in a scattered manner, large-area abnormality of the characteristic library caused by single machine faults is avoided, and the reliability of the characteristic library is improved.
In one possible embodiment, the feature library described above may be stored in an HDFS (Hadoop Distributed File System, distributed file system). HDFS is a distributed file system designed to fit on general purpose hardware. HDFS copies the features to three copies and stores them on different machines. In this way, feature anomalies due to single machine faults are further avoided, and feature safety is further improved.
In one possible embodiment, the features may be stored in partitions in the HDFS. The storage space in the feature library is divided into a plurality of areas, so that the features stored in each area are respectively managed, the feature management pertinence is improved, and the management efficiency is further improved. As a possible implementation manner, the storage space in the feature library may be partitioned according to different services, and the same partition stores features produced by the same service line. For example, the features may be stored to the partitions of the corresponding business lines according to the business line identifications included by each feature. The service line identifier may be a service line name, a code set for the service line, or the like. The service line may be set according to actual situations, and for example, for a financial enterprise, the service line may include: model generation business, air control business, credit loan business, and the like.
The user may issue a target feature request through the offline feature management platform. In one possible embodiment, the front-end page of the offline feature management platform may display function buttons, and as one possible implementation, may display query, add, delete, and modify buttons. Meanwhile, to-be-filled information can be set for each function button. As a possible implementation manner, the information that the query function needs to fill in can be set as service line identification, feature quantity and the like; the information to be filled in by the newly added function is the service line identification and the feature identification. The information that the delete function needs to fill in may include a feature identification of the feature that needs to be deleted. The information that the modification function needs to fill in may include the feature identification of the feature that needs to be modified and the modified feature information.
After the user logs in the offline feature management platform, the user can send out a target feature request by clicking a required function button and filling in corresponding information. The buttons are only for illustrating the offline feature management platform provided by the present invention, and do not limit the present invention specifically, and in other embodiments, other types of request sending entries may be set on the front page of the offline feature management platform, which is not limited specifically by the present invention.
After receiving the target feature request sent by the target user, the offline feature management platform can verify the target feature request through a verification module.
The target feature request may include target feature information, where the target feature information may include information that is filled in by the user when the request is issued, such as feature identifier, service line identifier, and department identifier. As described above, the target feature request is issued by the target user after logging into the offline feature management platform, and thus, the user information of the target user may also be included in the target feature request, where the user information may include the ID, the department, the name, and the like of the user. The verification module can determine whether the target user has the modification authority of the target feature contained in the target feature request based on the corresponding relation between the pre-stored feature and the user, wherein the modification authority comprises the authority of adding, modifying, deleting and the like. If the target user has the changing authority of the target feature, the target feature request can be sent to the management module, and the management module can change the feature library according to the target feature request. And the user can realize the automatic management of the feature library only by submitting a request through page interactive operation, and the problem of manual misoperation is avoided without directly touching data operation.
The corresponding relation between the features and the user can be set according to actual needs, and the corresponding relation between the features, departments and personnel can be set by way of example. In one possible embodiment, different permissions may also be set for different request types. The request types may include queries, additions, deletions, modifications, and the like.
For example, for the newly added type, the correspondence between the feature and the user and the department may be set, when the newly added feature request is received, it may be checked whether the user who sends the newly added feature request belongs to the department to which the feature belongs, and after verification, the feature may be bound to the user, that is, only the user as the responsible person of the feature has the modification and deletion authority of the feature. For the query type, a correspondence between the feature and the user group (RCC, role Control Center) may be set, and the user belonging to the same user group as the user uploading the feature may query the feature.
In one possible embodiment, as shown in fig. 2, fig. 2 is a schematic diagram of setting different management rights for each feature in the embodiment of the present invention, where the management rights may include storage, release, and reading. Storing refers to storing features in a distributed system, publishing refers to publishing features in an offline feature management platform for use by other users.
The storage feature permissions may be set for each feature first, and for users with feature storage directory permissions, they may be allowed to store the produced features in the HDFS. Specifically, the user output characteristics are stored on the HDFS, and the user only has the access right of the file under the own service line directory by splitting and storing the service lines, so that the safety problem caused by the leakage of the internal characteristics of departments is avoided.
And secondly, the authority of the characteristic platform can be set, and only the personnel with the access authority of the characteristic storage directory and the login authority of the platform can issue the characteristics on the characteristic platform, so that the random use of platform resources is avoided, and the safety of characteristic data is ensured.
And a user with the access authority of the feature can be set, and only the user with the RCC group to which the feature belongs can access the feature, and the access feature needs to verify the identity of the RCC group through the interface.
In one possible embodiment, the user may also have cross-user group access to features, i.e., the user may query for features of other user groups. But requires multiple authentications to the user prior to access.
As shown in fig. 3, fig. 3 is a schematic flow chart for verifying access rights of a user across a user group:
the verification module reads a target feature request of a user, sends the feature request to a user group leader client of a target feature to be accessed, a group leader client of a user group to which the user belongs and a preset security part client, and allows the user to access the target feature after receiving verification passing information returned by the three parties.
In one possible embodiment, a whitelist may also be preset, which may include request types that do not require manual review. As a possible implementation, the request types included in the whitelist may include queries as well as newly added types. In the case where the target request type included in the target feature request is the request type in the white list, there may be no need to manually audit the target feature request. In the case where the target request type included in the target feature request is not the request type in the white list, then a manual audit of the target feature request is required.
As a possible implementation manner, an auditor client may be preset, and if the request type of the target feature request is not in the preset white list, the target feature request may be sent to the preset client, so that the auditor can manually audit the target feature request. And the auditor can determine whether the user has the modification or deletion authority of the target feature according to the user information, the feature information contained in the target feature request and the user information, and if the user has the modification or deletion authority of the target feature, the audited information can be returned to the verification module. And after receiving the verification passing information, the verification module can send the target feature request to the management module. The number of the preset clients can be multiple.
By manually checking certain request types, the characteristics are prevented from being deleted or modified by mistake, and the safety of the characteristic library is further improved.
As shown in fig. 4, fig. 4 is a schematic diagram of the verification process when the target feature request is of a modification or deletion type:
the characteristic responsible person requests to delete or modify the characteristic, the group organization and the platform manager of the user group to which the characteristic responsible person belongs audit the request, and under the condition that audit passing information sent by two sides is received, the characteristic responsible person is determined to have the deleting/modifying authority of the characteristic, and the request can be sent to the management module.
In a possible embodiment, the verification module may be further configured to determine whether the feature update time accords with a preset update period, and output alarm information if the feature update time does not accord with the preset update period; and sending the target feature request to the management module if the update period is met.
In this embodiment, the feature update period, that is, the maximum time to increase the interval of new features, may be preset. If the target feature request is not sent out in the updating period, the related personnel can be informed to determine whether the feature output is abnormal or not, so that the feature output stability is ensured.
And after receiving the target feature request, the management module can operate the feature library according to the target feature request. In a possible embodiment, the target feature request may further include a storage path of the target feature, and the management module may change, such as delete, add, and modify, according to the target request type, according to the storage path of the target feature.
One or more configuration files can be preset for the feature library in the invention. For example, a configuration file may be preset for the feature library and each storage partition in the feature library, where a storage path of the feature, an RCC group to which a responsible person belongs, and so on may be recorded in the configuration file. After the feature library is changed according to the target feature request, the corresponding configuration file can be modified. Of course, a configuration file may be set for each machine, which is not particularly limited by the present invention.
For example, after adding the target feature, information of the target feature may be added to a configuration file corresponding to the feature library and a configuration file corresponding to the target feature partition, for example, an identifier, a storage location, the service identifier, a responsible person identifier, and information of the user group to which the target feature belongs may be added to the configuration file.
As described above, feature libraries are provided in HDFS, which typically manages data stored therein by hive (a data warehouse tool based on Hadoop for data extraction, transformation, loading, a mechanism that can store, query, and analyze large-scale data stored in Hadoop). Thus, in one possible embodiment, the management module may make changes to the feature library by hive in accordance with the target feature request.
In a possible embodiment, the offline feature management platform may further include a metadata base, where the metadata base is used to store description information of each feature, and the description information includes a storage location.
Correspondingly, before the feature library is changed according to the target feature request, the method further comprises: and adding the target feature request to the metadata base, and updating the metadata base according to the changed feature base by the management module.
The type of the metadata database can be selected according to actual needs, for example, mySQL, which is not particularly limited in the present invention. The metadata base is used for storing each feature and description information thereof, and the description information can comprise storage positions of the features, business lines of the features, production time of the features and the like.
In one possible embodiment, the target feature request may be added to the metadata repository upon receipt of the target feature request. For example, information such as the type of the target feature request, the target feature identifier, the target feature storage position and the like can be added into a metadata base to record the user feature request, so that feature change abnormality caused by the loss of the user feature request is avoided.
In one possible embodiment, the metadata base may synchronize the recorded target feature request to the configuration file of the feature library, so as to record the user feature request at the same time, and further avoid feature change abnormality caused by loss of the user feature request.
In one possible embodiment, in the case that the target feature request is a new feature, the feature in the configuration file may be copied, and the copied feature may be stored in a storage path included in the target feature request. In one possible embodiment, after a new feature is requested for a target feature to a feature library, the feature may also be published in an offline feature management platform for use by relevant personnel.
In a possible embodiment, in the case that the target feature request type is a query type, a storage location corresponding to the feature identifier may be determined from the metadata base according to the feature identifier included in the target feature request, so that the target feature is read from the location and returned to the user.
After the feature library is changed according to the target feature request, the changed feature information can be synchronized to the metadata library in addition to the configuration file, and the feature information and the description information of the features can be synchronized.
As shown in fig. 5, fig. 5 is a schematic flow chart of checking different target feature request types in the embodiment of the present invention:
after receiving a target feature request sent by a user after logging in the platform, the platform can verify the user authority based on the corresponding relation between the preset feature and the user under the condition that the target feature request is query feature information or newly added feature, and then the request information can be submitted to mysql, and the mysql can synchronize the target feature request to a configuration file.
In the case that the target feature request is a modified feature or a deleted feature, for example, basic information or a change threshold (an upper limit of fluctuation of feature data amounts of two adjacent time partitions) of the modified feature, or a request of deleting a partition, a table or the like, after submitting the request information to mysql, a manual audit mail may be sent to a manager. After the audit is passed, the target feature request may be synchronized from mysql to the configuration file. The time for synchronizing the target feature request from mysql into the configuration file may feel the actual need for selection, such as one point in the morning, twelve points, etc., or immediately after the audit is passed, which is not specifically limited by the present invention.
The management module may then make changes to the feature library, such as performing feature library entry, deletion, modification, etc., as requested in the configuration file. And synchronizing the changed information to the configuration file and mysql after the change is completed. For example, the newly added partition information may be updated into the configuration file as well as mysql.
As shown in fig. 6, fig. 6 is a schematic flow chart of the added features in the embodiment of the present invention, which specifically may include the following steps:
(1) and reading the characteristic information from the configuration file.
The offline feature management platform may synchronize the target feature request to a configuration file of the feature library after receiving the target feature request, thereby reading target feature information from the configuration file.
(2) And checking the characteristic data.
I.e. checking whether the target feature contained in the target feature request is normal, such as whether the feature format is correct, whether the feature data size is within a preset range, etc.
(3) And checking the authority of the feature responsible person to determine whether the user has the management authority of the target feature.
(4) And checking the characteristic updating period.
(5) Feature copy.
That is, after the feature verification is passed, the target feature in the target feature request is copied to the feature library, and the HDFS typically copies the feature into three copies, which are stored in different machines respectively.
(6) And (5) feature release.
For newly added features, the features may be published into an offline management platform for use by the user.
(7) And synchronizing the characteristic information to MySQL.
And (3) sending out alarm information when any one of the steps (1) - (7) is abnormal. The alarm information may be an alarm mail, an instant messaging information, etc. sent by a client of a platform manager.
As shown in fig. 7, fig. 7 is a flowchart illustrating an offline feature management platform according to an embodiment of the present invention:
specifically, after logging in the platform, the user sends a target feature request, where the target feature request may be a request for querying, adding, deleting, modifying, and the like. The offline feature management platform records the target feature request in MySQL and synchronizes the feature information to the configuration file. And then reading the target feature request in the configuration file, and changing the feature library according to the target feature request. Specifically, features may be added, deleted, or modified by hive. After the feature library is changed, the change content is synchronized to the configuration file and mysql.
By applying the embodiment of the invention, the characteristics are stored in the distributed clusters, so that the user group level authority control is realized, and the leakage of private information is avoided. And secondly, the characteristic data are uniformly stored in the HDFS multi-copy backup, and users uniformly manage and maintain the characteristic library through the platform, so that the consistency of the version of the characteristic data is ensured. Furthermore, the user can realize the automatic management of the characteristics of the feature library only by submitting the application form through page interactive operation, and the problem of manual misoperation is avoided without directly touching data operation. And through manual verification and an alarm mechanism, a user can acquire the warehousing condition of the features at the first time, and version information can be solved at the first time if abnormal conditions of the features are met.
Based on the same inventive concept, the invention also provides an offline feature management method, which is applied to the offline feature management platform, as shown in fig. 8, and the method may include:
s801, under the condition that a target feature request sent by a target user is received, determining whether the target user has the management authority of a target feature included in the target feature request;
s802, under the condition that the target user has the management authority, changing a feature library according to the target feature request; the feature library is used for storing features in a distributed mode;
s803, updating the configuration file of the feature library according to the changed feature library.
In a possible embodiment, the determining whether the target user has the management authority of the target feature included in the target feature request includes:
and determining whether the target user belongs to a target user group corresponding to the target feature included in the target feature request, and determining that the target user has the management authority of the target feature under the condition that the target user belongs to the target user group.
In one possible embodiment, the method further comprises:
determining the target feature request type;
transmitting the target feature request to the management module under the condition that the target feature request type is a preset white list type;
transmitting the target feature request to a preset client under the condition that the target feature request type is not the preset white list type;
and under the condition that the audit passing information returned by the preset client is received, sending the target feature request to the management module.
The exemplary embodiment of the invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores a computer program executable by the at least one processor for causing the electronic device to perform a method according to an embodiment of the invention when executed by the at least one processor.
The exemplary embodiments of the present invention also provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is for causing the computer to perform a method according to an embodiment of the present invention.
The exemplary embodiments of the invention also provide a computer program product comprising a computer program, wherein the computer program, when being executed by a processor of a computer, is for causing the computer to perform a method according to an embodiment of the invention.
With reference to fig. 9, a block diagram of an electronic device 900 that may be a server or a client of the present invention will now be described, which is an example of a hardware device that may be applied to aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 9, the electronic device 900 includes a computing unit 901 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 902 or a computer program loaded from a storage unit 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904.
A number of components in the electronic device 900 are connected to the I/O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device. The output unit 907 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 908 may include, but is not limited to, magnetic disks, optical disks. The communication unit 909 allows the electronic device 900 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth (TM) devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 901 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 901 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the respective methods and processes described above. For example, in some embodiments, the offline feature management method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 900 via the ROM 902 and/or the communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the offline feature management method by any other suitable means (e.g., by means of firmware).
Program code for carrying out methods of the present invention may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
Claims (10)
1. An offline feature management platform, the platform comprising: the system comprises a verification module, a management module and a feature library;
the feature library is used for storing all the features in a distributed manner;
the verification module is used for determining whether the target user has the management authority of the target feature included in the target feature request or not under the condition that the target feature request sent by the target user is received; sending the target feature request to a management module under the condition that the target user has the management authority;
the management module is used for changing the feature library according to the target feature request; and updating the configuration file of the feature library according to the changed feature library.
2. The platform of claim 1, wherein the determining whether the target user has administrative rights for the target feature included in the target feature request comprises:
and determining whether the target user belongs to a target user group corresponding to the target feature included in the target feature request, and determining that the target user has the management authority of the target feature under the condition that the target user belongs to the target user group.
3. The platform of claim 1, wherein the platform is configured to support the platform,
the verification module is further used for determining the target feature request type; transmitting the target feature request to the management module under the condition that the target feature request type is a preset white list type; transmitting the target feature request to a preset client under the condition that the target feature request type is not the preset white list type; and under the condition that the audit passing information returned by the preset client is received, sending the target feature request to the management module.
4. The platform of claim 1, wherein the platform is configured to support the platform,
the verification module is further used for determining whether the characteristic updating time accords with a preset updating period or not, and outputting alarm information under the condition that the characteristic updating time does not accord with the preset updating period; and under the condition that the characteristic updating time accords with the preset updating period, sending the target characteristic request to the management module.
5. The platform of claim 1, further comprising:
a metadata base for storing description information of each of the features, the description information including a storage location;
before the feature library is changed according to the target feature request, the method further comprises the following steps: adding the target feature request to the metadata repository,
and the management module is also used for updating the metadata base according to the changed feature base.
6. An offline feature management method, characterized by being applied to the offline feature management platform of claim 1; the method comprises the following steps:
under the condition that a target feature request sent by a target user is received, determining whether the target user has the management authority of a target feature included in the target feature request;
under the condition that the target user has the management authority, changing a feature library according to the target feature request; the feature library is used for storing features in a distributed mode;
and updating the configuration file of the feature library according to the changed feature library.
7. The method of claim 6, wherein the determining whether the target user has administrative rights for the target feature included in the target feature request comprises:
and determining whether the target user belongs to a target user group corresponding to the target feature included in the target feature request, and determining that the target user has the management authority of the target feature under the condition that the target user belongs to the target user group.
8. The method according to claim 1, wherein the method further comprises:
determining the target feature request type;
transmitting the target feature request to the management module under the condition that the target feature request type is a preset white list type;
transmitting the target feature request to a preset client under the condition that the target feature request type is not the preset white list type;
and under the condition that the audit passing information returned by the preset client is received, sending the target feature request to the management module.
9. An electronic device, comprising:
a processor; and
a memory in which a program is stored,
wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the method according to any of claims 6-8.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 6-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311685890.XA CN117668811A (en) | 2023-12-08 | 2023-12-08 | Offline feature management platform, offline feature management method, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311685890.XA CN117668811A (en) | 2023-12-08 | 2023-12-08 | Offline feature management platform, offline feature management method, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117668811A true CN117668811A (en) | 2024-03-08 |
Family
ID=90076782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311685890.XA Pending CN117668811A (en) | 2023-12-08 | 2023-12-08 | Offline feature management platform, offline feature management method, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117668811A (en) |
-
2023
- 2023-12-08 CN CN202311685890.XA patent/CN117668811A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9959306B2 (en) | Partition-based index management in hadoop-like data stores | |
| JP2022529967A (en) | Extracting data from the blockchain network | |
| US20130304697A1 (en) | System and method for cross-platform synchronization of data objects in a multi-tenant environment | |
| US20200050380A1 (en) | Predictive forecasting and data growth trend in cloud services | |
| EP4207688A1 (en) | Asynchronous bookkeeping method and apparatus for blockchain, medium, and electronic device | |
| CN112801607A (en) | Management service platform and construction method | |
| US11995453B2 (en) | Method and apparatus for generating image file and computer-readable storage medium | |
| CN111327613B (en) | Distributed service authority control method and device and computer readable storage medium | |
| CN111324606B (en) | Data slicing method and device | |
| CN111338571B (en) | Task processing method, device, equipment and storage medium | |
| CN112685499A (en) | Method, device and equipment for synchronizing process data of work service flow | |
| US20220269495A1 (en) | Application deployment in a computing environment | |
| CN112995112A (en) | Resource management method of cross-cloud management platform | |
| US11093527B2 (en) | Framework for continuous processing of a set of documents by multiple software applications | |
| CN112256760B (en) | Data prediction method and device, computer equipment and storage medium | |
| CN105844171B (en) | Method and device for file synchronization control | |
| CN112800066A (en) | Index management method, related device and storage medium | |
| CN117668811A (en) | Offline feature management platform, offline feature management method, electronic equipment and storage medium | |
| CN114282210A (en) | Sandbox automatic construction method and system, computer equipment and readable storage medium | |
| CN114925044A (en) | Data synchronization method, device and equipment based on cloud storage and storage medium | |
| CN114911575A (en) | ClickHouse cluster management method and related equipment | |
| CN113449035B (en) | Data synchronization method, device, computer equipment and readable storage medium | |
| CN112699093A (en) | File storage method and equipment supporting hybrid multi-cloud | |
| JP7409735B1 (en) | Operational design document creation device | |
| AU2020452837A1 (en) | Provision of remote application action feed cards |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |