CN117194991A - High-dimensional data recommendation system and method based on GPU cluster - Google Patents
High-dimensional data recommendation system and method based on GPU cluster Download PDFInfo
- Publication number
- CN117194991A CN117194991A CN202311452396.9A CN202311452396A CN117194991A CN 117194991 A CN117194991 A CN 117194991A CN 202311452396 A CN202311452396 A CN 202311452396A CN 117194991 A CN117194991 A CN 117194991A
- Authority
- CN
- China
- Prior art keywords
- data
- gpu
- training
- module
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 119
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000004364 calculation method Methods 0.000 claims description 43
- 238000013500 data storage Methods 0.000 claims description 19
- 238000004891 communication Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000007726 management method Methods 0.000 claims description 7
- 238000012986 modification Methods 0.000 claims description 6
- 230000004048 modification Effects 0.000 claims description 6
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data processing, in particular to a high-dimensional data recommendation system and method based on a GPU cluster, which are characterized in that training data in a data training task are subjected to data splitting at a client, and a generated original data module sequence corresponding to the training data is sent to the GPU cluster; according to the original data module sequence of the corresponding training data, obtaining data training task characteristics, and generating a computing power container of the corresponding GPU cluster by the GPU cluster according to the data training task characteristics; distributing each data module in the original data module sequence in GPU units contained in a computing power container of the corresponding GPU cluster according to computing power occupation of the original data module sequence of the corresponding training data uploaded by the client; processing according to the data training task, generating data after data training, and generating recommended data according to the data after data training. By the technical scheme provided by the invention, the data processing efficiency can be improved.
Description
Technical Field
The invention relates to the field of data processing, in particular to a high-dimensional data recommendation system and method based on a GPU cluster.
Background
With the rapid development of the internet, the demands of users for personalized recommendations are increasing. To meet this demand, high-dimensional data recommendation methods based on GPU clusters have been developed. However, in implementing this approach, many technical challenges need to be addressed. First, the processing of high-dimensional data is a difficult problem. Due to the extremely high data dimensions, conventional data processing methods often fail to process these data efficiently. Therefore, development of a new data processing method is required to accommodate the characteristics of high-dimensional data. Second, the use of GPU clusters also presents some technical challenges. Because of the special architecture of the GPU cluster, the data processing method needs to be optimized for the GPU cluster to ensure the high efficiency and accuracy of data processing. In addition, the choice of recommendation algorithm is also a critical issue. Different recommendation algorithms are suitable for different data types and user requirements. Therefore, the most suitable recommendation algorithm needs to be selected according to the actual situation.
Aiming at the problems, the data processing method based on the GPU cluster high-dimensional data recommendation needs to develop a complete technical scheme, which comprises links of data preprocessing, feature extraction, model training, model evaluation and the like. Meanwhile, optimization is required to be performed on the framework of the GPU cluster so as to improve the data processing efficiency and accuracy. In the data preprocessing stage, a proper data dimension reduction method is needed to convert high-dimension data into low-dimension data so as to reduce the complexity and the calculated amount of the data. At the same time, data cleaning and feature extraction are also required to obtain more accurate and useful data. In the model training stage, a proper recommendation algorithm needs to be selected, and optimization is performed according to the characteristics of the GPU cluster. In addition, model tuning and parameter adjustment are needed to improve the accuracy and generalization capability of the model. In the model evaluation stage, the effect of the recommended model needs to be evaluated by adopting proper evaluation indexes and methods. In summary, the data processing method based on GPU cluster high-dimensional data recommendation needs to solve many technical challenges, including links such as high-dimensional data processing, GPU cluster optimization, recommendation algorithm selection, and model evaluation.
Therefore, how to process data that needs to be recommended by high-dimensional data is a subject that needs to be studied by technicians in the current industry.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a high-dimensional data recommendation method based on a GPU cluster, which comprises the following steps:
step one, data splitting is carried out on training data in a data training task at a client to obtain an original data module sequence corresponding to the training data, and the client sends the generated original data module sequence corresponding to the training data to a GPU cluster;
step two, the GPU cluster obtains an original data module sequence corresponding to the training data, and obtains data training task characteristics according to the original data module sequence corresponding to the training data, and the GPU cluster generates an algorithm force container corresponding to the GPU cluster according to the data training task characteristics;
step three, according to the calculation power occupation of the original data module sequence of the corresponding training data uploaded by the client, if the calculation power occupation of the original data module sequence of the corresponding training data is within the capacity range of the calculation power container of the corresponding GPU cluster, distributing each data module in the original data module sequence in the GPU units contained in the calculation power container of the corresponding GPU cluster, and entering step four; if the computing power occupation of the original data module sequence corresponding to the training data is not in the capacity range of the computing power container corresponding to the GPU cluster, entering a step five;
step four, the GPU unit manager wakes up and sends the distributed data modules to the corresponding GPU units, the GPU units process the distributed data modules according to the data training tasks, after the data processing is completed, data after the data training is generated, the GPU units are released until all GPU units contained in the computing power container of the GPU cluster are completed, and step six is entered;
step five, the GPU unit manager calls GPU units corresponding to the calculation force difference values to the calculation force containers of the GPU clusters in the calculation force containers of the corresponding GPU clusters according to the calculation force difference values, generates corrected calculation force containers of the corresponding GPU clusters, distributes data training tasks to the calculation force containers of the corresponding GPU clusters, carries out data processing on an original data module sequence of the corresponding training data, generates data after the data training is completed, releases the GPU units, and enters step six;
step six, the GPU cluster uploads the original data module sequence corresponding to the training data to the data storage module, meanwhile, the GPU cluster generates basic data state information according to the information of the original data module sequence corresponding to the training data, the serial number is N, and recommended data are generated according to the data after the data training.
Further, the data splitting is performed on the training data in the data training task at the client to obtain an original data module sequence corresponding to the training data, which includes:
splitting training data in a data training task into a plurality of data modules at a client, numbering the data modules in sequence, generating a data module index table according to the numbers, packaging the data module index table and the first-ordered data module to generate a header file, and generating an original data module sequence of corresponding training data by the header file and the rest data modules.
Further, the GPU cluster obtains an original data module sequence corresponding to the training data, obtains data training task features according to the original data module sequence corresponding to the training data, and generates an computing power container corresponding to the GPU cluster according to the data training task features, including: the GPU cluster obtains the number of data modules according to the head files in the obtained original data module sequences corresponding to the training data, and obtains the number of required computing units according to the number of the data modules, wherein the number of the required computing units is the characteristic of the data training task;
obtaining the required number of GPU units according to the required number of the calculation units and the number of the unit calculation units of the GPU units, and generating a power calculation container corresponding to the GPU cluster by calling the required number of the GPU units in the GPU cluster according to the required number of the GPU units.
Further, the GPU cluster uploads the original data module sequence corresponding to the training data to the data storage module, and simultaneously generates basic data state information according to the information of the original data module sequence corresponding to the training data, the sequence number is N, and generates recommended data according to the data after the data training, including:
performing data operation on the original data module in the computing power container corresponding to the GPU cluster, performing state information modification on the basis of the basic data state information of the GPU cluster according to the content of the data operation, generating basic data state information with a sequence number added with one, and updating the basic data state information with the sequence number added with one into the data storage module to form a basic data state information sequence with the basic data state information; when the GPU unit accesses data, the data storage module firstly matches the original data according to the serial number of the basic data state information stored by the GPU cluster, then matches the corresponding basic data state information according to the number of the serial number plus one, and carries out corresponding data operation on the original data according to the content of the data operation in the corresponding basic data state information to generate recommended data.
The high-dimensional data recommendation system based on the GPU cluster is applied to the high-dimensional data recommendation method based on the GPU cluster, and comprises a GPU cluster module, a client, a data storage module, a communication module and a GPU management unit;
the client and the GPU cluster module are respectively in communication connection with the communication module; and the data storage module and the GPU management unit are respectively connected with the GPU cluster module.
The beneficial effects of the invention are as follows: according to the technical scheme provided by the invention, the data to be processed can be segmented and distributed to the specific GPU units, and the data processing efficiency is improved.
Drawings
FIG. 1 is a flow diagram of a high-dimensional data recommendation method based on GPU clusters;
fig. 2 is a schematic diagram of a high-dimensional data recommendation system based on GPU clusters.
Detailed Description
The technical solution of the present invention will be described in further detail with reference to the accompanying drawings, but the scope of the present invention is not limited to the following description.
For the purpose of making the technical solution and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the particular embodiments described herein are illustrative only and are not intended to limit the invention, i.e., the embodiments described are merely some, but not all, of the embodiments of the invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention. It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The features and capabilities of the present invention are described in further detail below in connection with the examples.
As shown in fig. 1, the high-dimensional data recommendation method based on the GPU cluster includes the following steps:
step one, data splitting is carried out on training data in a data training task at a client to obtain an original data module sequence corresponding to the training data, and the client sends the generated original data module sequence corresponding to the training data to a GPU cluster;
step two, the GPU cluster obtains an original data module sequence corresponding to the training data, and obtains data training task characteristics according to the original data module sequence corresponding to the training data, and the GPU cluster generates an algorithm force container corresponding to the GPU cluster according to the data training task characteristics;
step three, according to the calculation power occupation of the original data module sequence of the corresponding training data uploaded by the client, if the calculation power occupation of the original data module sequence of the corresponding training data is within the capacity range of the calculation power container of the corresponding GPU cluster, distributing each data module in the original data module sequence in the GPU units contained in the calculation power container of the corresponding GPU cluster, and entering step four; if the computing power occupation of the original data module sequence corresponding to the training data is not in the capacity range of the computing power container corresponding to the GPU cluster, entering a step five;
step four, the GPU unit manager wakes up and sends the distributed data modules to the corresponding GPU units, the GPU units process the distributed data modules according to the data training tasks, after the data processing is completed, data after the data training is generated, the GPU units are released until all GPU units contained in the computing power container of the GPU cluster are completed, and step six is entered;
step five, the GPU unit manager calls GPU units corresponding to the calculation force difference values to the calculation force containers of the GPU clusters in the calculation force containers of the corresponding GPU clusters according to the calculation force difference values, generates corrected calculation force containers of the corresponding GPU clusters, distributes data training tasks to the calculation force containers of the corresponding GPU clusters, carries out data processing on an original data module sequence of the corresponding training data, generates data after the data training is completed, releases the GPU units, and enters step six;
step six, the GPU cluster uploads the original data module sequence corresponding to the training data to the data storage module, meanwhile, the GPU cluster generates basic data state information according to the information of the original data module sequence corresponding to the training data, the serial number is N, and recommended data are generated according to the data after the data training.
The data splitting is performed on training data in a data training task at the client to obtain an original data module sequence corresponding to the training data, and the data splitting comprises the following steps:
splitting training data in a data training task into a plurality of data modules at a client, numbering the data modules in sequence, generating a data module index table according to the numbers, packaging the data module index table and the first-ordered data module to generate a header file, and generating an original data module sequence of corresponding training data by the header file and the rest data modules.
The GPU cluster obtains an original data module sequence corresponding to training data, obtains data training task characteristics according to the original data module sequence corresponding to the training data, and generates an algorithm force container corresponding to the GPU cluster according to the data training task characteristics, and comprises the following steps: the GPU cluster obtains the number of data modules according to the head files in the obtained original data module sequences corresponding to the training data, and obtains the number of required computing units according to the number of the data modules, wherein the number of the required computing units is the characteristic of the data training task;
obtaining the required number of GPU units according to the required number of the calculation units and the number of the unit calculation units of the GPU units, and generating a power calculation container corresponding to the GPU cluster by calling the required number of the GPU units in the GPU cluster according to the required number of the GPU units.
The GPU cluster uploads an original data module sequence corresponding to training data to a data storage module, generates basic data state information according to information of the original data module sequence corresponding to the training data, has a sequence number of N, generates recommended data according to data after data training, and comprises the following steps:
performing data operation on the original data module in the computing power container corresponding to the GPU cluster, performing state information modification on the basis of the basic data state information of the GPU cluster according to the content of the data operation, generating basic data state information with a sequence number added with one, and updating the basic data state information with the sequence number added with one into the data storage module to form a basic data state information sequence with the basic data state information; when the GPU unit accesses data, the data storage module firstly matches the original data according to the serial number of the basic data state information stored by the GPU cluster, then matches the corresponding basic data state information according to the number of the serial number plus one, and carries out corresponding data operation on the original data according to the content of the data operation in the corresponding basic data state information to generate recommended data.
As shown in fig. 2, the GPU cluster-based high-dimensional data recommendation system applies the GPU cluster-based high-dimensional data recommendation method, and the GPU cluster-based high-dimensional data recommendation system comprises a GPU cluster module, a client, a data storage module, a communication module and a GPU management unit;
the client and the GPU cluster module are respectively in communication connection with the communication module; and the data storage module and the GPU management unit are respectively connected with the GPU cluster module.
The GPU cluster module is used for generating an algorithm force container corresponding to the GPU cluster according to the data training task characteristics;
the client splits the training data in the data training task to obtain an original data module sequence corresponding to the training data, and sends the generated original data module sequence corresponding to the training data to the GPU cluster.
The GPU management unit is used for distributing the data modules and dispatching the GPU units, and dispatching the GPU units to the generated computing power containers corresponding to the GPU clusters.
The data storage module is used for storing data generated after the computing power container corresponding to the GPU cluster performs data operation on the original data module.
The GPU cluster module further comprises a GPU cluster node and a calculation container generation module; the GPU cluster node comprises a plurality of GPU units, and the power calculation container generation module is used for generating power calculation containers corresponding to the GPU clusters according to the required power calculation.
The foregoing is merely a preferred embodiment of the invention, and it is to be understood that the invention is not limited to the form disclosed herein but is not to be construed as excluding other embodiments, but is capable of numerous other combinations, modifications and environments and is capable of modifications within the scope of the inventive concept, either as taught or as a matter of routine skill or knowledge in the relevant art. And that modifications and variations which do not depart from the spirit and scope of the invention are intended to be within the scope of the appended claims.
Claims (5)
1. The high-dimensional data recommendation method based on the GPU cluster is characterized by comprising the following steps of:
step one, data splitting is carried out on training data in a data training task at a client to obtain an original data module sequence corresponding to the training data, and the client sends the generated original data module sequence corresponding to the training data to a GPU cluster;
step two, the GPU cluster obtains an original data module sequence corresponding to the training data, and obtains data training task characteristics according to the original data module sequence corresponding to the training data, and the GPU cluster generates an algorithm force container corresponding to the GPU cluster according to the data training task characteristics;
step three, according to the calculation power occupation of the original data module sequence of the corresponding training data uploaded by the client, if the calculation power occupation of the original data module sequence of the corresponding training data is within the capacity range of the calculation power container of the corresponding GPU cluster, distributing each data module in the original data module sequence in the GPU units contained in the calculation power container of the corresponding GPU cluster, and entering step four; if the computing power occupation of the original data module sequence corresponding to the training data is not in the capacity range of the computing power container corresponding to the GPU cluster, entering a step five;
step four, the GPU unit manager wakes up and sends the distributed data modules to the corresponding GPU units, the GPU units process the distributed data modules according to the data training tasks, after the data processing is completed, data after the data training is generated, the GPU units are released until all GPU units contained in the computing power container of the GPU cluster are completed, and step six is entered;
step five, the GPU unit manager calls GPU units corresponding to the calculation force difference values to the calculation force containers of the GPU clusters in the calculation force containers of the corresponding GPU clusters according to the calculation force difference values, generates corrected calculation force containers of the corresponding GPU clusters, distributes data training tasks to the calculation force containers of the corresponding GPU clusters, carries out data processing on an original data module sequence of the corresponding training data, generates data after the data training is completed, releases the GPU units, and enters step six;
step six, the GPU cluster uploads the original data module sequence corresponding to the training data to the data storage module, meanwhile, the GPU cluster generates basic data state information according to the information of the original data module sequence corresponding to the training data, the serial number is N, and recommended data are generated according to the data after the data training.
2. The GPU cluster-based high-dimensional data recommendation method of claim 1, wherein the splitting of the training data in the data training task at the client to obtain the original data module sequence corresponding to the training data comprises:
splitting training data in a data training task into a plurality of data modules at a client, numbering the data modules in sequence, generating a data module index table according to the numbers, packaging the data module index table and the first-ordered data module to generate a header file, and generating an original data module sequence of corresponding training data by the header file and the rest data modules.
3. The high-dimensional data recommendation method based on the GPU cluster according to claim 2, wherein the GPU cluster obtains an original data module sequence corresponding to training data, obtains data training task features according to the original data module sequence corresponding to the training data, and generates an algorithm container corresponding to the GPU cluster according to the data training task features, and the method comprises the steps of: the GPU cluster obtains the number of data modules according to the head files in the obtained original data module sequences corresponding to the training data, and obtains the number of required computing units according to the number of the data modules, wherein the number of the required computing units is the characteristic of the data training task;
obtaining the required number of GPU units according to the required number of the calculation units and the number of the unit calculation units of the GPU units, and generating a power calculation container corresponding to the GPU cluster by calling the required number of the GPU units in the GPU cluster according to the required number of the GPU units.
4. The method for recommending high-dimensional data based on GPU clusters according to claim 1, wherein the GPU clusters upload the original data module sequence corresponding to the training data to the data storage module, and the GPU clusters generate the basic data status information according to the information of the original data module sequence corresponding to the training data, with the sequence number of N, and generate recommended data according to the data after the data training, comprising:
performing data operation on the original data module in the computing power container corresponding to the GPU cluster, performing state information modification on the basis of the basic data state information of the GPU cluster according to the content of the data operation, generating basic data state information with a sequence number added with one, and updating the basic data state information with the sequence number added with one into the data storage module to form a basic data state information sequence with the basic data state information; when the GPU unit accesses data, the data storage module firstly matches the original data according to the serial number of the basic data state information stored by the GPU cluster, then matches the corresponding basic data state information according to the number of the serial number plus one, and carries out corresponding data operation on the original data according to the content of the data operation in the corresponding basic data state information to generate recommended data.
5. The high-dimensional data recommendation system based on the GPU cluster is characterized by comprising a GPU cluster module, a client, a data storage module, a communication module and a GPU management unit, wherein the high-dimensional data recommendation method based on the GPU cluster is applied to any one of claims 1-4;
the client and the GPU cluster module are respectively in communication connection with the communication module; and the data storage module and the GPU management unit are respectively connected with the GPU cluster module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311452396.9A CN117194991B (en) | 2023-11-03 | 2023-11-03 | High-dimensional data recommendation system and method based on GPU cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311452396.9A CN117194991B (en) | 2023-11-03 | 2023-11-03 | High-dimensional data recommendation system and method based on GPU cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117194991A true CN117194991A (en) | 2023-12-08 |
| CN117194991B CN117194991B (en) | 2024-02-13 |
Family
ID=88994524
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311452396.9A Active CN117194991B (en) | 2023-11-03 | 2023-11-03 | High-dimensional data recommendation system and method based on GPU cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194991B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209896A (en) * | 2019-06-06 | 2019-09-06 | 江苏户传科技有限公司 | A kind of product quality tracing system based on artificial intelligence big data platform |
| CN112241321A (en) * | 2020-09-24 | 2021-01-19 | 北京影谱科技股份有限公司 | Computing power scheduling method and device based on Kubernetes |
| CN112668650A (en) * | 2020-12-30 | 2021-04-16 | 上海电气集团股份有限公司 | Industrial data model generation method, system, device and medium |
| CN112988390A (en) * | 2021-03-22 | 2021-06-18 | 上海超级计算中心 | Calculation power resource allocation method and device |
| WO2022002068A1 (en) * | 2020-06-29 | 2022-01-06 | 中兴通讯股份有限公司 | Data processing method, system and device and storage medium |
| CN114145006A (en) * | 2020-06-12 | 2022-03-04 | 华为技术有限公司 | Scheduling method and device of artificial intelligence resources, storage medium and chip |
| CN114269445A (en) * | 2019-08-26 | 2022-04-01 | 辉达公司 | Content recommendation using one or more neural networks |
| US20220240408A1 (en) * | 2021-01-22 | 2022-07-28 | Nvidia Corporation | Static data center power balancing and configuration |
| CN115543615A (en) * | 2022-09-29 | 2022-12-30 | 上海商汤科技开发有限公司 | Resource allocation method and device, electronic equipment and storage medium |
| US20230115163A1 (en) * | 2021-12-31 | 2023-04-13 | Beijing Baidu Netcom Science Technology Co., Ltd. | Method for processing data, and electronic device, storage medium and program product |
| CN116069500A (en) * | 2022-12-20 | 2023-05-05 | 中国电信股份有限公司 | Model training task processing method and device, electronic equipment and readable medium |
| CN116450355A (en) * | 2023-04-21 | 2023-07-18 | 重庆长安汽车股份有限公司 | Multi-cluster model training method, device, equipment and medium |
| CN116954929A (en) * | 2023-09-20 | 2023-10-27 | 四川并济科技有限公司 | Dynamic GPU scheduling method and system for live migration |
-
2023
- 2023-11-03 CN CN202311452396.9A patent/CN117194991B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209896A (en) * | 2019-06-06 | 2019-09-06 | 江苏户传科技有限公司 | A kind of product quality tracing system based on artificial intelligence big data platform |
| CN114269445A (en) * | 2019-08-26 | 2022-04-01 | 辉达公司 | Content recommendation using one or more neural networks |
| CN114145006A (en) * | 2020-06-12 | 2022-03-04 | 华为技术有限公司 | Scheduling method and device of artificial intelligence resources, storage medium and chip |
| WO2022002068A1 (en) * | 2020-06-29 | 2022-01-06 | 中兴通讯股份有限公司 | Data processing method, system and device and storage medium |
| CN112241321A (en) * | 2020-09-24 | 2021-01-19 | 北京影谱科技股份有限公司 | Computing power scheduling method and device based on Kubernetes |
| CN112668650A (en) * | 2020-12-30 | 2021-04-16 | 上海电气集团股份有限公司 | Industrial data model generation method, system, device and medium |
| US20220240408A1 (en) * | 2021-01-22 | 2022-07-28 | Nvidia Corporation | Static data center power balancing and configuration |
| CN112988390A (en) * | 2021-03-22 | 2021-06-18 | 上海超级计算中心 | Calculation power resource allocation method and device |
| US20230115163A1 (en) * | 2021-12-31 | 2023-04-13 | Beijing Baidu Netcom Science Technology Co., Ltd. | Method for processing data, and electronic device, storage medium and program product |
| CN115543615A (en) * | 2022-09-29 | 2022-12-30 | 上海商汤科技开发有限公司 | Resource allocation method and device, electronic equipment and storage medium |
| CN116069500A (en) * | 2022-12-20 | 2023-05-05 | 中国电信股份有限公司 | Model training task processing method and device, electronic equipment and readable medium |
| CN116450355A (en) * | 2023-04-21 | 2023-07-18 | 重庆长安汽车股份有限公司 | Multi-cluster model training method, device, equipment and medium |
| CN116954929A (en) * | 2023-09-20 | 2023-10-27 | 四川并济科技有限公司 | Dynamic GPU scheduling method and system for live migration |
Non-Patent Citations (2)
| Title |
|---|
| JAY H. PARK等: "HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism", 《2020 USENIX ANNUAL TECHNICAL CONFERENCE》, pages 307 - 321 * |
| 沙章利: "基于AWS GPU集群的协同过滤算法的研究及应用", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 03, pages 138 - 7909 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117194991B (en) | 2024-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10354201B1 (en) | Scalable clustering for mixed machine learning data | |
| CN103886047B (en) | Towards the online recommendation method of distribution of stream data | |
| CN109739939A (en) | The data fusion method and device of knowledge mapping | |
| CN111611488B (en) | Information recommendation method and device based on artificial intelligence and electronic equipment | |
| US10268749B1 (en) | Clustering sparse high dimensional data using sketches | |
| CN102567312A (en) | Machine translation method based on distributive parallel computation framework | |
| US20230030265A1 (en) | Object processing method and apparatus, storage medium, and electronic device | |
| CN112396462A (en) | Crowd circling method and device based on Clickhouse | |
| CN117194991B (en) | High-dimensional data recommendation system and method based on GPU cluster | |
| CN113628043B (en) | Complaint validity judging method, device, equipment and medium based on data classification | |
| CN114119289A (en) | Method and device for processing comprehensive energy monitoring data | |
| CN114817178A (en) | Industrial Internet data storage method, system, storage medium and electronic equipment | |
| CN107844536A (en) | The methods, devices and systems of application program selection | |
| CN112214602A (en) | Text classification method and device based on humor, electronic equipment and storage medium | |
| CN109669777B (en) | Industrial internet big data element demand service providing method and system | |
| CN113778681B (en) | Data processing method and device based on cloud computing and storage medium | |
| CN110795559A (en) | Data processing method and device for customer service question answering | |
| Xiao et al. | Field-wise embedding size search via structural hard auxiliary mask pruning for click-through rate prediction | |
| CN114298319B (en) | Determination method and device for joint learning contribution value, electronic equipment and storage medium | |
| CN111562990B (en) | Lightweight serverless computing method based on message | |
| CN113076450B (en) | Determination method and device for target recommendation list | |
| CN112214683B (en) | Mixed recommendation model processing method, system and medium based on heterogeneous information network | |
| CN115048422A (en) | Process recommendation method, device, equipment and storage medium | |
| CN114997687A (en) | Resource scheduling method and device, storage medium and electronic equipment | |
| CN116107630B (en) | Multi-platform adaptation method for big data operation and maintenance monitoring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |