WO2023207665A1 - Data processing method and related device - Google Patents

Abstract

A data processing method, relating to the field of artificial intelligence. Multiple linear transformations are performed on a feature block input into a head to respectively obtain a first amplitude value, a first phase value, a second amplitude value, and a second phase value, wherein the first amplitude value is used as an amplitude of an element in a Q matrix, the first phase value is used as a phase of the element in the Q matrix, the second amplitude value is used as an amplitude of an element in a K matrix, and the second phase value is used as a phase of the element in the K matrix. According to the present application, the amplitude and the phase of the element in the Q matrix and the amplitude and the phase of the element in the K matrix are respectively generated for the feature block input into the head, and the Q matrix and the K matrix are parameterized into a more complex form, so as to improve expression capability of a feature, so that a relationship between different feature blocks can be better modeled.

Description

A data processing method and related equipment

This application claims priority to the U.S. patent application filed with the U.S. Patent Office on April 29, 2022, with application number 17/733,758 and the invention title "METHOD AND DEVICE FOR PROCESSING DATA BASED ON MULTI-LAYER PERCEPTRONS", and in 2022 The priority of the Chinese patent application filed with the China Patent Office on July 30, 2019, with the application number 202210912635.3 and the invention title "A data processing method and related equipment", the entire content of which is incorporated into this application by reference.

Technical field

This application relates to the field of artificial intelligence, and in particular, to a data processing method and related equipment.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

With the continuous development of artificial intelligence technology, natural language human-computer interaction systems that enable humans and machines to interact through natural language are becoming more and more important. For humans and machines to interact through natural language, the system needs to be able to recognize the specific meaning of human natural language. Usually, the system identifies the specific meaning of the sentence by extracting key information from the natural language sentence.

The transformer structure has powerful semantic expression capabilities and can capture long text dependencies. Since it was proposed, it has significantly surpassed previous models in a series of natural language processing tasks represented by translation. Language models based on the transformer structure have also achieved very good results in question and answer systems, voice assistants and other fields.

The transformer structure network includes a multi-head self-attention module. However, the attention head in the multi-head self-attention module only calculates the relationship between different feature blocks within the real number domain, and the modeling capability is insufficient.

Contents of the invention

This application provides a data processing method that generates the amplitude and phase of the elements in the Q matrix and the amplitude and phase of the elements in the K matrix respectively for the feature block token input into the head, thereby increasing the expression ability of the features and enabling better Model the relationship between different feature blocks.

In the first aspect, this application provides a data processing method applied to the attention head head in the neural network based on the attention mechanism. The method includes: performing multiple linear operations on the feature blocks input to the head. Transform to obtain the first amplitude, the second phase value, the second amplitude and the second phase value respectively; according to the first numerical value and the first phase value, obtain the Q matrix; the first amplitude is used As the amplitude of the element in the Q matrix, the first phase value is used as the phase of the element in the Q matrix; according to the second amplitude and the second phase value, the K matrix is obtained; the second The amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the phase of the element in the K matrix; V calculated according to the Q matrix, the K matrix and the head matrix, get the head's output.

In the existing implementation, the transformation matrix Q and transformation matrix K are used to linearly transform the feature block token input into the head, and the Q matrix and K matrix are obtained respectively. The elements in the Q matrix and K matrix are all real numbers. For Features have limited expressive power. In this application, the amplitude and phase of the elements in the Q matrix and the amplitude and phase of the elements in the K matrix can be generated respectively for the feature block token input into the head. In this way, both the Q matrix and the K matrix can be parameterized into complex forms, thereby increasing the expressive ability of features and better modeling the relationship between different feature blocks.

In a possible implementation, obtaining the output of the head based on the Q matrix, the K matrix and the V matrix calculated in the head includes: comparing the Q matrix and the K matrix Perform correlation calculation to obtain a first attention matrix, and the elements in the first attention matrix are complex numbers; map the elements in the first attention matrix to real numbers to obtain a second attention matrix; according to The second attention matrix and the V matrix calculated in the head are used to obtain the output of the head.

In one implementation, correlation calculation can be performed on the Q matrix and the K matrix to obtain the first attention matrix. Since the Q matrix and the K matrix themselves are numbers in the complex domain, the first attention matrix The elements in the force matrix are complex numbers.

In one implementation, when calculating the correlation based on the Q matrix and the K matrix, the phase difference existing between the elements in the Q matrix and the K matrix may cause the amplitude to be modulated, for example, with a small phase difference Correlated features may be enhanced, while features associated with large phase differences may be reduced, similar to interference phenomena caused by coherent light.

In one implementation, the dot product result of qi and kj can be directly determined as the correlation degree. Since the correlation degree calculated above needs to be weighted with the V matrix, the elements in the first attention matrix can be mapped is a real number, obtain the second attention matrix, and obtain the output of the head according to the second attention matrix and the V matrix calculated in the head.

In one implementation, the dot multiplication result can be divided by a constant, and then the softmax operation is performed. Since the softmax operation needs to be performed in the real number domain, the result of the operation between the Q matrix and the K matrix (that is, the object of the softmax operation ) is a complex number. Therefore, in the embodiment of the present application, the result of the operation between the Q matrix and the K matrix (the first attention matrix) can be mapped to the real number domain.

In one implementation, the elements in the first attention matrix include real parts and imaginary parts. When mapping the elements in the first attention matrix to real numbers, the elements in the first attention matrix can be The numerical value of the real part and the numerical value of the imaginary part are fused to obtain a fusion result, and the fusion result is a real number.

For example, the fusion may include: a summation operation; or, finding the modulus length of a complex number.

In a possible implementation, the first amplitude, the first phase value, the second amplitude and the second phase value are obtained by performing multiple linear transformations on the elements of the feature block input to the head. , including: performing multiple linear transformations on the feature blocks input into the head through a linear programming layer; the linear programming layer includes a first weight, a second weight, a third weight and a fourth weight, the first The weight, the second weight, the third weight and the fourth weight are trainable parameters; wherein the first weight is used to linearly transform the feature block to obtain the first image value; the second weight is used to linearly transform the feature block to obtain the first phase value; the third weight is used to linearly transform the feature block to obtain the second phase value. value; the fourth weight is used to The feature block is linearly transformed to obtain the second phase value.

In this application, each query query and keyword key use two linear transformation layers to generate amplitude and phase respectively, so that the generated query and key can express richer information.

In a possible implementation, the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data.

In a second aspect, this application provides a data processing method, which method includes:

By performing multiple linear transformations on the first feature block in the feature map, the first amplitude and first phase values are obtained respectively;

According to the first amplitude and the first phase value, a second feature block is obtained; the first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used as the phase of the elements in the second feature block; map the elements in the second feature block to real numbers to obtain a third feature block; input the third feature block into the convolution layer.

For the feature map that needs to be processed by the convolution layer, each feature block in the feature map (the first feature block is taken as an example in the embodiment of this application) can be linearly transformed multiple times to obtain the first amplitude value and the first phase value respectively. The first amplitude value is used as the amplitude of the element in the second feature block, and the first phase value is used as the phase of the element in the second feature block.

Different from existing methods, the feature block in this application uses multiple linear transformations to generate amplitude and phase respectively, so that the generated optimized feature block can express richer information. And in the final output layer of the network, the features are transformed into the real number domain through feature transformation to obtain practical meaningful model output.

In a possible implementation, mapping the elements in the second feature block to real numbers includes:

The values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers are fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion includes: a concatenation operation (concat).

In a possible implementation, the first amplitude and the first phase value are obtained respectively by performing multiple linear transformations on the first feature block in the feature map, including:

Through the linear programming layer, multiple linear transformations are performed on the first feature block in the feature map; the linear programming layer includes a first weight and a second weight, and the first weight and the second weight are trainable parameters. ;in,

The first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude;

The second weight is used to linearly transform the first feature block in the feature map to obtain the first phase value.

In a possible implementation, the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data.

In the third aspect, this application provides a data processing device applied to the attention head in a neural network based on the attention mechanism. The device includes:

A linear transformation module is used to perform multiple linear transformations on the feature blocks input to the head to obtain respectively a first amplitude value, a second phase value, a second amplitude value and a second phase value;

Attention calculation module, used to obtain the Q matrix according to the first numerical value and the first phase value; the first amplitude value is used as the amplitude value of the element in the Q matrix, and the first phase value is used to As the phase of the elements in the Q matrix;

According to the second amplitude and the second phase value, a K matrix is obtained; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the element in the K matrix. the phase of an element;

According to the Q matrix, the K matrix and the V matrix calculated in the head, the output of the head is obtained.

In a possible implementation, the attention calculation module is specifically used to:

Perform correlation calculation on the Q matrix and the K matrix to obtain a first attention matrix, where the elements in the first attention matrix are complex numbers;

Map the elements in the first attention matrix to real numbers to obtain a second attention matrix;

According to the second attention matrix and the V matrix calculated in the head, the output of the head is obtained.

In a possible implementation, the elements in the first attention matrix include real parts and imaginary parts, and the attention calculation module is specifically used to:

The values of the real part and the value of the imaginary part of the elements in the first attention matrix are fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion includes:

sum operation; or,

Find the modular length of a complex number.

In a possible implementation, the linear transformation module is specifically used for:

Through the linear planning layer, multiple linear transformations are performed on the feature blocks input into the head; the linear planning layer includes a first weight, a second weight, a third weight and a fourth weight. The first weight, the The second weight, the third weight and the fourth weight are trainable parameters; wherein,

The first weight is used to linearly transform the feature block to obtain the first amplitude;

The second weight is used to linearly transform the feature block to obtain the first phase value;

The third weight is used to linearly transform the feature block to obtain the second amplitude;

The fourth weight is used to linearly transform the feature block to obtain the second phase value.

In a possible implementation, the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data.

In a fourth aspect, this application provides a data processing device, which includes:

The linear transformation module is used to perform multiple linear transformations on the first feature block in the feature map to obtain the first Amplitude and first phase values;

According to the first amplitude and the first phase value, a second feature block is obtained; the first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used as the The phase of the elements in the second feature block;

Map elements in the second feature block to real numbers to obtain a third feature block;

The convolution module is used to input the third feature block into the convolution layer.

In a possible implementation, the linear transformation module is specifically used for:

The values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers are fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion includes:

Concatenation operation (concat).

In a possible implementation, the linear transformation module is specifically used for:

Through the linear programming layer, multiple linear transformations are performed on the first feature block in the feature map; the linear programming layer includes a first weight and a second weight, and the first weight and the second weight are trainable parameters. ;in,

The first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude;

The second weight is used to linearly transform the first feature block in the feature map to obtain the first phase value.

In a possible implementation, the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data.

In the fifth aspect, embodiments of the present application provide an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the above-mentioned first aspect and Any optional method thereof, and the above second aspect and any optional method thereof.

In a sixth aspect, embodiments of the present application provide a training device, which may include a memory, a processor, and a bus system. The memory is used to store programs, and the processor is used to execute programs in the memory to perform the above-mentioned first aspect and Any optional method thereof, and the above second aspect and any optional method thereof.

In a seventh aspect, embodiments of the present application provide a computer-readable storage medium that stores a computer program that, when run on a computer, causes the computer to execute the first aspect and any one of the above-mentioned aspects. Optional methods, as well as the above-mentioned second aspect and any optional method thereof.

In an eighth aspect, embodiments of the present application provide a computer program that, when run on a computer, causes the computer to execute the above-mentioned first aspect and any optional method thereof, as well as the above-mentioned second aspect and any optional method thereof. method of selection.

In a ninth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, for example, sending or processing data involved in the above methods; Or, information. In a possible design, the chip system further includes a memory, and the memory is used to store necessary program instructions and data for executing the device or training the device. The chip system may be composed of chips, or may include chips and other discrete devices.

Description of the drawings

Figure 1 is a structural schematic diagram of the main framework of artificial intelligence;

Figure 2 shows a natural language processing system;

Figure 3a shows another natural language processing system;

Figure 3b is a schematic diagram of a system;

Figure 4 is a schematic diagram of related equipment for natural language processing provided by an embodiment of the present application;

Figure 5A is a schematic diagram of the architecture of a transformer layer;

Figure 5B is a schematic diagram of the architecture of a convolutional neural network;

Figure 5C is a schematic diagram of the architecture of a convolutional neural network;

Figure 6 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application;

Figure 7 is a schematic structural representation of a neural network model in an embodiment of the present application;

Figure 8 is a schematic diagram of the operation of an attention head;

Figure 9 is a schematic diagram of the operation of a head provided by an embodiment of the present application;

Figure 10 is a fusion diagram provided by an embodiment of the present application;

Figure 11 is a schematic diagram of an application architecture provided by an embodiment of the present application;

Figure 12 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application;

Figure 13 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application;

Figure 14 is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

Figure 15 is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

Figure 16 is a schematic structural diagram of an execution device provided by an embodiment of the present application;

Figure 17 is a schematic structural diagram of the training equipment provided by the embodiment of the present application;

Figure 18 is a schematic structural diagram of a chip provided by an embodiment of the present application.

Detailed ways

The embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific embodiments of the present invention and are not intended to limit the present invention.

The embodiments of the present application are described below with reference to the accompanying drawings. Persons of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

The terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances, and are merely a way of distinguishing objects with the same attributes in describing the embodiments of the present application. Furthermore, the terms "include" and "having" and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, product or apparatus comprising a series of elements need not be limited to those elements, but may include not explicitly other elements specifically listed or inherent to such processes, methods, products or equipment.

First, the overall workflow of the artificial intelligence system is described. Please refer to Figure 1. Figure 1 shows a structural schematic diagram of the artificial intelligence main framework. The following is from the "intelligent information chain" (horizontal axis) and "IT value chain" ( vertical axis) two Dimension elaborates on the above artificial intelligence theme framework. Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (providing and processing technology implementation) to the systematic industrial ecological process.

(1)Infrastructure

Infrastructure provides computing power support for artificial intelligence systems, enables communication with the external world, and supports it through basic platforms. Communicate with the outside through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA, etc.); the basic platform includes distributed computing framework and network and other related platform guarantees and support, which can include cloud storage and Computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, which are provided to smart chips in the distributed computing system provided by the basic platform for calculation.

(2)Data

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3)Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formal information to perform machine thinking and problem solving based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability

After the data is processed as mentioned above, some general capabilities can be formed based on the results of further data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image processing. identification, etc.

(5) Intelligent products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.

This application can be applied to natural language processing, image processing and other fields in the field of artificial intelligence. The following will introduce multiple application scenarios that have been implemented into products.

In order to better understand the solutions of the embodiments of the present application, possible application scenarios of the embodiments of the present application are briefly introduced below with reference to Figures 1 to 3a.

Figure 2 shows a natural language processing system, which includes user equipment and data processing equipment. Among them, user equipment includes smart terminals such as mobile phones, personal computers, or information processing centers. User equipment is from However, as the initiator of language data processing, as the initiator of language question and answer or query requests, the user usually initiates the request through the user device.

The above-mentioned data processing equipment may be a cloud server, a network server, an application server, a management server, and other equipment or servers with data processing functions. The data processing equipment receives query statements/voice/text, etc. from the smart terminal through an interactive interface, and then performs language data processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor for data processing. , and feedback the processing results to the user device. The memory in the data processing device can be a general term, including local storage and a database that stores historical data. The database can be on the data processing device or on other network servers.

In the natural language processing system shown in Figure 2, the user device can receive the user's instructions. For example, the user device can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device can respond to the piece of text obtained by the user device. The text executes natural language processing applications (such as natural language generation, text classification, text reasoning, named entity recognition, translation, etc.), thereby obtaining the processing results of the corresponding natural language processing application for the text (such as predicted word results, classification results) , inference results, named entity recognition results, translation results, etc.).

Taking natural language generation as an example, natural language generation (natural language generation) can also be called text prediction task or natural language synthesis task. It refers to the task of generating missing text or subsequent text given a piece of text. . Natural language generation is widely used in search engines, input methods and other scenarios. It can predict the user's next input based on the user's input of part of the text, which can greatly improve the user's efficiency in using the product. In addition, it can also detect missing text. text to be restored.

Exemplarily, in the embodiment of the present application, the user equipment can receive a piece of text data input by the user, where the text data includes known words and words to be predicted. The words to be predicted are not visible, and only the location of the words to be predicted in the text data is known. location, and then the user device can initiate a request to the data processing device (the request carries text data), so that the data processing device predicts the word to be predicted in the text data, thereby obtaining the word to be predicted, and feeds the word to be predicted to the user equipment.

For example, the user device can receive a piece of text data input by the user, and then initiate a request to the data processing device, so that the data processing device performs entity classification on the piece of text data, thereby obtaining the entity classification result for the piece of text data, and The entity classification results are fed back to the user device;

For example, the user device can receive a piece of text data input by the user (the text data is Chinese text), and then initiate a request to the data processing device, so that the data processing device translates the piece of text data into English, thereby obtaining the text data for the piece of text data. English translation, and feedback the English translation to the user device.

Figure 3a shows another natural language processing system. In Figure 3a, the user device directly serves as a data processing device. The user device can directly receive input from the user and process it directly by the hardware of the user device itself. The specific process is as follows Figure 2 is similar, please refer to the above description, and will not be repeated here.

FIG. 4 is a schematic diagram of a natural language processing related device 300 provided by an embodiment of the present application.

The user equipment in Figure 2 and Figure 3a can be the local device 301 or the local device 302 in Figure 4, and the data processing device in Figure 2 can be the execution device 310 in Figure 4, where the data storage system 350 can To store the data to be processed by the execution device 310, the data storage system 350 can be integrated on the execution device 310, or can be set up on the cloud or other network servers.

The processors in Figures 2 and 3a can perform data training/machine learning/deep learning through neural network models or other models, and use the model finally trained or learned on the data to execute natural language processing applications (such as natural language generation) on text data , text classification, sequence annotation, reading comprehension, text generation, text reasoning, translation, etc.) to obtain corresponding processing results.

Furthermore, in computer vision, neural network architectures are often used to handle various tasks such as image classification, object detection, and semantic segmentation.

The system architecture provided by the embodiment of the present application will be introduced in detail below with reference to Figure 3b. Figure 3b is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in Figure 3b, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550 and a data collection system 560.

The execution device 510 includes a computing module 511, an I/O interface 512, a preprocessing module 513 and a preprocessing module 514. The target model/rule 501 may be included in the calculation module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

Data collection device 560 is used to collect training data.

Among them, in the task of natural language synthesis, the training data can be text data with missing text and complete text data corresponding to the text data with missing text.

Among them, in the translation task, the training data can include but is not limited to parallel corpus, monolingual corpus, etc.

Parallel corpus refers to bilingual or multilingual corpus (that is, text data with annotations) composed of original text and its parallel corresponding target text. The original text and target text have the same semantics and there is correspondence between text units. relation. For example, the original text is "This trip needs careful planning", and the corresponding English text is "The trip needs careful planning", then "This trip needs careful planning" and "The trip needs careful planning" can be regarded as a set of parallels. Corpus, this group of parallel corpora is a Chinese-English parallel language pair. The original text "This trip needs careful planning" can be regarded as the source material of this group of parallel corpora, and the translated text "The trip needs careful planning" can be regarded as this group of parallel corpora. The target corpus of the corpus. Among them, travel may correspond to trip.

In addition, "This trip needs careful planning" can be regarded as a monolingual corpus, and "The trip needs careful planning" can also be regarded as a monolingual corpus.

In computer vision-related tasks, training data can be images, labels corresponding to the images, enhanced images, etc.

After collecting the training data, the data collection device 560 stores the training data into the database 530, and the training device 520 trains to obtain the target model/rule 501 based on the training data maintained in the database 530.

Among them, the training device 520 trains the pretrained language model (PLM) in the embodiment of the present application based on the training data maintained in the database 530 to obtain the target model/rule 501.

It should be noted that in actual applications, the training data maintained in the database 530 may not necessarily be collected by the data collection device 560, but may also be received from other devices. In addition, it should be noted that the training device 520 does not necessarily perform training of the target model/rules 501 based entirely on the training data maintained by the database 530. It may also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation of this application. Limitations of Examples.

The target model/rules 501 trained according to the training device 520 can be applied to different systems or devices, such as to the execution device 510 shown in Figure 3b. The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, augmented reality (AR)/virtual reality (VR) devices, vehicle-mounted terminals, etc., or servers or clouds, etc. In Figure 3b, the execution device 510 is configured with an input/output (I/O) interface 512 for data interaction with external devices. The user can input data to the I/O interface 512 through the client device 540.

The preprocessing module 513 and the preprocessing module 514 are used to perform preprocessing according to the input data received by the I/O interface 512 (for example, obtaining the position of the known data unit and the data unit to be predicted in the target data, or generating attention information, etc. preprocessing process). It should be understood that there may be no preprocessing module 513 and 514 or only one preprocessing module. When the preprocessing module 513 and the preprocessing module 514 do not exist, the computing module 511 can be directly used to process the input data.

When the execution device 510 preprocesses input data, or when the calculation module 511 of the execution device 510 performs calculations and other related processes, the execution device 510 can call data, codes, etc. in the data storage system 550 for corresponding processing. , the data, instructions, etc. obtained by corresponding processing can also be stored in the data storage system 550.

Finally, the I/O interface 512 presents the processing results to the client device 540, thereby providing them to the user.

In the situation shown in FIG. 3 b , the user can manually set the input data, and the "manually set input data" can be operated through the interface provided by the I/O interface 512 . In another case, the client device 540 can automatically send input data to the I/O interface 512. If requiring the client device 540 to automatically send the input data requires the user's authorization, the user can set corresponding permissions in the client device 540. The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form may be display, sound, action, etc. The client device 540 can also be used as a data collection terminal to collect the input data of the input I/O interface 512 and the output results of the output I/O interface 512 as new sample data, and store them in the database 530. Of course, it is also possible to collect without going through the client device 540. Instead, the I/O interface 512 directly uses the input data input to the I/O interface 512 and the output result of the output I/O interface 512 as a new sample as shown in the figure. The data is stored in database 530.

It is worth noting that Figure 3b is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3b, the data The storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510.

It should be understood that the above execution device 510 can also be deployed in the client device 540.

Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms involved in the embodiments of the present application and related concepts such as neural networks are first introduced below.

(1)Neural network

The neural network can be composed of neural units. The neural unit can refer to an operation unit that takes xs (ie, input data) and intercept 1 as input. The output of the operation unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to transform the neural network. The input signal in the unit is converted into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(2)Transformer layer

Referring to Figure 5A, Figure 5A is a schematic diagram of the architecture of a transformer layer. As shown in Figure 5A, the neural network includes an embedding layer and at least one transformer layer. At least one transformer layer can be N transformer layers (N is an integer greater than 0). Among them, each transformer layer includes successively adjacent attention layers, addition and normalization (add&norm) layers, feed forward layers, and addition and normalization layers. In the embedding layer, the current input is embedded to obtain multiple embedding vectors; in the attention layer, P input vectors are obtained from the upper layer of the first transformer layer, and any of the P input vectors are The first input vector is the center. Based on the correlation between each input vector within the preset attention window range and the first input vector, the intermediate vector corresponding to the first input vector is obtained. In this way, P input vectors are determined. Corresponding P intermediate vectors; in the pooling layer, the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained by the last transformer layer in the transformer layer are used as features of the current input express.

(3) attention mechanism

The attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensation to increase the precision of observation in some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. . The attention mechanism can quickly extract important features of sparse data and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement of the attention mechanism, which reduces the dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of the attention mechanism can be rewritten as the following formula:

Among them, Lx=||Source|| represents the length of Source. The meaning of the formula is to imagine that the constituent elements in Source are composed of a series of data pairs. At this time, given a certain element Query in the target Target, by calculating the Query and Based on the similarity or correlation of each Key, the weight coefficient of each Key's corresponding Value is obtained, and then the Value is weighted and summed to obtain the final Attention value. So essentially the Attention mechanism is a weighted summation of the Value values of the elements in the Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring most of the unimportant information. The process of focusing is reflected in the calculation of the weight coefficient. The greater the weight, the more focused it is on its corresponding Value value. That is, the weight represents the importance of the information, and the Value is its corresponding information. The self-attention mechanism can be understood as internal Attention (intra attention). The Attention mechanism occurs between the Target element Query and all elements in the Source. The self-attention mechanism refers to between the internal elements of the Source or between the internal elements of the Target. The Attention mechanism that occurs can also be understood as the attention calculation mechanism in the special case of Target=Source. The specific calculation process is the same, but the calculation object has changed.

(4) Natural language processing (NLP)

Natural language is human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding and extracting information from text data in an intelligent and efficient way. By using NLP and its components, we can manage very large pieces of text data, or perform a large number of automated tasks, and solve a variety of problems, such as automatic summarization (automatic summarization), machine translation (MT), Named entity recognition (NER), relationship extraction (RE), information extraction (IE), sentiment analysis, speech recognition (speech recognition), question answering system (question answering), topic segmentation, etc. .

(5) Multilayer perceptron (MLP)

Multi-layer perceptron is a forward-structured artificial neural network architecture composed of multiple node layers that can transform input vectors into output vectors.

(6)Quantum mechanics

Quantum mechanics is one of the fundamental pillars of modern physics and the foundation of many modern disciplines.

(7) Feature block (token)

A piece of feature or data.

(8) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal. In the convolutional layer of a convolutional neural network, a neuron can be connected to only some of the neighboring layer neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as extracting features in a way that is independent of location. The convolution kernel can be formalized as a matrix of random size. During the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

CNN is a very common neural network. The structure of CNN will be introduced in detail below with reference to Figure 5B. As mentioned in the previous introduction to the basic concepts, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. The deep learning architecture refers to the algorithm of machine learning. Multiple levels of learning at different levels of abstraction. As a deep learning architecture, CNN is a feed-forward artificial neural network. Each neuron in the feed-forward artificial neural network can respond to the image input into it.

As shown in Figure 5B, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

Convolutional layer/pooling layer 220:

Convolutional layer:

As shown in Figure 5B, the convolution layer/pooling layer 220 may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, and layer 223 is a convolution layer. Product layer, 224 is a pooling layer, 225 is a convolution layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolution layers, 223 is a pooling layer, and 224 and 225 are convolutions. layer, 226 is the pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.

The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

The convolution layer 221 can include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually predefined. During the convolution operation on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. ...This depends on the value of the step size) to complete the process of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a convolved output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns) are applied, That is, multiple matrices of the same type. The output of each weight matrix is stacked to form the depth dimension of the convolution image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to remove unnecessary noise in the image. Perform blurring, etc. The multiple weight matrices have the same size (row × column), and the feature maps extracted by the multiple weight matrices with the same size are also the same size. The extracted multiple feature maps with the same size are then merged to form a convolution operation. output.

The weight values in these weight matrices require a large amount of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, thereby allowing the convolutional neural network 200 to make correct predictions. .

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features; as the convolutional neural network As the depth of the network 200 deepens, the features extracted by subsequent convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. In each layer 221-226 as shown at 220 in Figure 5B, there can be a layer of convolution layer followed by a layer of The pooling layer can also be a multi-layer convolution layer followed by one or more pooling layers. During image processing, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The max pooling operator can take the pixel with the largest value in a specific range as the result of max pooling. In addition, just like the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the size of the image. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 230:

After being processed by the convolutional layer/pooling layer 220, the convolutional neural network 200 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 220 will only extract features and reduce the parameters brought by the input image. However In order to generate the final output information (required class information or other related information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate the output of one or a set of required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in Figure 5B), and the parameters contained in the multiple hidden layers may be based on the relevant training data of the specific task type. Obtained by pre-training, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc...

After the multi-layer hidden layer in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to categorical cross entropy and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (the propagation from the direction 210 to 240 in Figure 5B is forward propagation) is completed, the back propagation (the propagation from the direction 240 to 210 in Figure 5B is back propagation) will Start updating the weight values and biases of each layer mentioned above to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in Figure 5B is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, only Including part of the network structure shown in FIG. 5B , for example, the convolutional neural network used in the embodiment of the present application may only include an input layer 210 , a convolutional layer/pooling layer 220 and an output layer 240 .

It should be noted that the convolutional neural network 100 shown in Figure 5B is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, as The multiple convolutional layers/pooling layers shown in Figure 5C are parallel, and the respectively extracted features are input to the fully connected layer 230 for processing.

Referring to Figure 6, Figure 6 is a schematic diagram of a data processing method provided by an embodiment of the present application. The data processing method provided by an embodiment of the present application can be applied to terminal devices such as mobile phones, tablets, notebook computers, and smart wearable devices. on the server, and can also be applied on the server. As shown in Figure 6, a data processing method provided by the embodiment of the present application includes:

601. By performing multiple linear transformations on the feature blocks input to the head, obtain the first amplitude value, the second phase value, the second amplitude value and the second phase value respectively.

In some embodiments, a computing device (eg, terminal device, smartphone, laptop, server, etc.) acquires multiple feature blocks (tokens) for a piece of data (eg, an image, a piece of audio/video, or a piece of context data). Each token can be generated as a piece of data that serves as the basic unit for processing by a computing device. Therefore, each token includes information (eg, characteristics or content) associated with part of the data. For example, a token can be defined as an image patch containing 16×16 pixels in the input image. Computing devices can implement neural network architectures based on attention mechanisms to process multiple tokens.

In a possible implementation, a neural network based on the attention mechanism can process input data. The input data can be a piece of data, and a piece of data can be audio data, video data, image data or context data. When a piece of data is divided into n When there are n tokens, the characteristic information contained in the data fragment can be distributed on n tokens. The token set (represented by Z) associated with the input data fragment can be expressed as Z = [z ₁ , z ₂ ,..., z _n ], z _j represents the jth token, and j and n are integers.

In a possible implementation, after obtaining the input data, each data of the input data can be The fragments are embedded and processed to obtain multiple feature blocks (tokens). The tokens here can also be called embedding vectors.

Taking the input data as text data as an example, in a possible implementation, the embedding layer may include an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each data unit in the unmasked data unit in the current input, thereby obtaining the word vector of each data unit in the unmasked data unit (for example, you can represents semantic information). At the position coding layer, the position of each data unit in the current input can be obtained and a position vector can be generated for the position of each data unit in the unmasked data unit.

In some examples, the position information of each of the unmasked data units in the data sequence may be the absolute position of each of the unmasked data units in the data sequence. Taking the current input as "What number should I pay back the Huabei?" for example, the position of "number" can be represented as the first digit, the position of "number" can be represented as the second digit,... In some examples, the position of each of the unmasked data units in the data sequence may be the relative position of each of the unmasked data units in the data sequence. Still taking the current input as "what number should I pay back Huabei" as an example, the position of "what number" can be expressed as before "number", and the position of "number" can be expressed as after "what number" and before "should",... …. When the word vector and position vector of each data unit in the unmasked data unit in the current input are obtained, the position vector of each data unit in the unmasked data unit and the corresponding word vector can be obtained. Fusion to obtain the embedding vector of each data unit in the unmasked data units. It should be understood that the fusion method can be an addition operation of the position vector and the corresponding word vector, or other operations, and the specific fusion method is not limited here. Embedding vectors can be represented as embedding matrices with preset dimensions. The number of embedding vectors can be set to M, and the default dimension is H dimension. Then the embedding vector can be expressed as an M×H embedding matrix.

In a possible implementation, multiple feature blocks can be input into the transformer layer, specifically into each attention head (head) of the multi-head attention heads in the transformer layer. Next, the transformer layer is introduced:

Referring to Figure 7, Figure 7 is a schematic structural diagram of a neural network model in an embodiment of the present application. As shown in Figure 7, the pre-trained language model may include an embedding layer and multiple transformer layers connected in sequence (which may also be called a feature extraction network in this embodiment). As those skilled in the art understand, the transformer model is mostly used to perform natural language processing NLP tasks. It should be understood that the structure in Figure 7 is just an example, and the number of transformer layers can be set as needed. For example, you can set up only one transformer layer, or you can set up more transformer layers. The neural network model determines the feature vector corresponding to the current node based on the N output vectors obtained from each transformer layer.

The specific working process of each layer is described below.

About the embedding layer:

In the embedding layer (or called the complex embedding layer), the current input is embedded to obtain multiple feature vectors (this vector is a complex vector (or it can be called a complex-valued vector), optionally, this vector is fixed a complex-valued vector of dimensions). The core feature of the transformer model lies in its unique attention mechanism. When processing natural language, such as a sentence, the transformer model uses this attention mechanism to assign different attention coefficients to each word vector in the sentence, thereby more comprehensively considering the impact of the context on each word in the sentence. The embedding layer obtains N embedding vectors X _l based on the node characteristics and position encoding of each node in the current sequence. The attention layer is connected to the embedding layer, and N embedding vectors are obtained from the embedding layer as input vectors. Based on the correlation between each input vector in the N input vectors, each input vector is synthesized to obtain N output vectors, which are output to Subsequent transformer layer. The transformer layer gets the previous layer's The output is used as an input vector and performs similar operations to the previous transformer layer.

About feature extraction network:

In one possible implementation, the feature extraction network can include multiple transformer layers.

Referring to Figures 7 and 8, both Figures 7 and 8 show a structural diagram of a transformer layer. The transformer layer of each neural network in the embodiment of the present application can refer to the structure shown in Figure 8, as shown in Figure 8 As shown, the transformer layer includes sequentially adjacent multi-head attention layers, summation and normalization (add&norm) layers, feed forward (feed forward) layers, and summation and normalization layers.

Among them, the multi-head attention layer obtains N input vectors X _l from its upper layer, which can also be expressed as a matrix It can also be expressed as matrix Y. It can be understood that when the multi-head attention layer is a layer directly connected to the embedding layer, such as the t1ransformer layer directly connected to the embedding layer in Figure 7, the input vector it obtains is the embedding vector output by the embedding layer; when the multi-head attention layer The layer is the multi-head attention layer included in the subsequent transformer layer. For example, in Figure 7, the multi-head attention layer included in the transformer layer directly connected to the previous level transformer layer. The input vector obtained is the output vector of the previous level transformer layer. . In the multi-head attention layer, the MHA layer based on multi-head attention (MHA) includes multiple attention heads (Head 1, Head 2, ..., Head N as shown in Figure 7).

Figure 8 is a schematic diagram of the operation of the attention head, which shows how the attention head transforms the input matrix X into the output matrix Y. As shown in Figure 8, in the existing implementation, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V are respectively used to each of the N input vectors <X1, X2,...,XN> Xi is transformed to obtain the first intermediate vector (q vector), the second intermediate vector (k vector) and the third intermediate vector (v vector) corresponding to each input vector. Operationally, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V can be used to linearly transform the input matrix X composed of N input vectors, respectively, to obtain the Q matrix, K matrix and V of the input matrix. Matrix, and then split the matrix separately to obtain the q vector, k vector and v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, based on the first intermediate vector (q vector, qi) corresponding to the i-th input vector and each second intermediate vector (k vector, kj) corresponding to each input vector Xj The dot product operation is used to determine each degree of correlation (α _i,j ) between the i-th input vector Xi and each input vector Xj. Although the dot multiplication result of qi and kj can also be directly determined as the correlation degree, more classically, the dot multiplication result is divided by a constant, and then a softmax operation is performed, and the operation result is used as the correlation degree of the input vectors Xi and Xj, Right now:

Therefore, each correlation degree αi,j between the i-th input vector Xi and each input vector Xj can be used as a weighting factor to perform a weighted combination on the third intermediate vector (v vector, vj) corresponding to each input vector The i-th combination vector Ci corresponding to the i input vector Xi:

Therefore, the vector sequence <C1, C2,...,CN>, or matrix C, of N combined vectors corresponding to N input vectors can be obtained. Based on this combined vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combined vectors can be directly used as N output vectors, that is, Yi=Ci. At this time, the output matrix Y is the combined vector matrix C, which can also be written as:

In the existing implementation, the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V are respectively used to perform the input matrix X composed of N input vectors (that is, the feature block token input into the head). Linear transformation obtains the Q matrix, K matrix and V matrix of the input matrix respectively. The elements in the Q matrix and K matrix are all real numbers, and the ability to express features is limited.

Different from the above-mentioned first transformation matrix Q and second transformation matrix K, in this application, the amplitude and phase of the elements in the Q matrix and the amplitude and phase of the elements in the K matrix can be generated respectively for the feature block token input into the head. In this way, both the Q matrix and the K matrix can be parameterized into complex forms, thereby increasing the expressive ability of features and better modeling the relationship between different feature blocks.

Referring to Figure 9, Figure 9 is a schematic diagram of the operation of an attention head in an embodiment of the present application. By performing multiple linear transformations on the elements of the feature block X input into the head, the first amplitude, the first phase value, the second amplitude and the second phase value are obtained respectively. The first amplitude value and the first phase value can be used to construct the elements in the Q matrix, and the second amplitude value and the second phase value can be used to construct the elements in the K matrix.

In a possible implementation, multiple linear transformations can be performed on the feature blocks input to the head through a linear programming layer; the linear programming layer includes a first weight Wq1, a second weight Wq2, and a third weight Wk1 And the fourth weight Wk2, the first weight Wq1, the second weight Wq2, the third weight Wk1 and the fourth weight Wk2 are trainable parameters; wherein the first weight is used to linearly transform the feature block , to obtain the first amplitude; the second weight is used to linearly transform the feature block to obtain the first phase value; the third weight is used to linearly transform the feature block , to obtain the second amplitude; the fourth weight is used to linearly transform the feature block to obtain the second phase value.

Through the above method, the elements in the obtained Q matrix and K matrix can be expressed as:

Among them, i is the imaginary part indicator, which satisfies i ² =-1, |·| represents the absolute value operation, ⊙ is element-wise multiplication, and the amplitude |z _j | is the real-valued feature, representing the content of each feature block. is a periodic function with unit norm. θ _j represents the phase, which represents the position of the current state within a cycle. Each feature block It can be represented in the complex domain with both phase and amplitude information. This kind of feature block represented by a wave function can also be called a quantum feature block.

602. Obtain the Q matrix according to the first numerical value and the first phase value; the first amplitude value is used as the amplitude of the element in the Q matrix, and the first phase value is used as the Q matrix phase of the elements.

603. According to the second amplitude and the second phase value, obtain the K matrix; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the K The phase of the elements in the matrix.

In the multi-head self-attention network, there are three-way features: query, keyword key and value. Both the query query and the keyword key can be represented as quantum feature blocks through a linear mapping layer, thereby calculating the attention matrix in the complex domain and better modeling the relationship between different feature blocks.

The specific implementation process is as follows:

Given X as the input feature, is the linear transformation layer, and are quantum feature blocks, and their calculation method is:

Wq1 and Wq2 together form the linear transformation layer of the query, and Wk1 and Wk2 form the linear transformation layer of the keyword key. They transform the input features into different feature spaces, making the quantum feature block have strong expressive ability. Most of the existing queries and keys are generated using a linear transformation layer, which limits their expressive capabilities; different from existing methods, each query and key in this application use two linear transformation layers to generate amplitudes respectively. value and phase, so that the generated query and key can express richer information.

604. Obtain the output of the head according to the Q matrix, the K matrix and the V matrix calculated in the head.

In one implementation, the correlation between the input feature blocks can be calculated based on the Q matrix and the K matrix. Specifically, it can be based on the q vector (qi) corresponding to the i-th input vector Xi in the input feature block. The dot multiplication operation of each k vector (kj) corresponding to the j-th input vector Xj determines each correlation degree between the i-th input vector Xi and each input vector Xj. Although the dot product result of qi and kj can also be directly determined as the correlation degree (that is, the attention matrix, or the complex-valued attention matrix), more classically, the dot product result is divided by a constant and then the softmax operation is performed. , use the operation result as the correlation between the input vectors Xi and Xj (that is, the attention matrix, or the complex-valued attention matrix), that is:

In one implementation, correlation calculation can be performed on the Q matrix and the K matrix to obtain the first attention matrix. Since the Q matrix and the K matrix themselves are numbers in the complex domain, the first attention matrix The elements in the force matrix are complex numbers.

In one implementation, when calculating the correlation based on the Q matrix and the K matrix, the phase difference existing between the elements in the Q matrix and the K matrix may cause the amplitude to be modulated, for example, with a small phase difference Correlated features may be enhanced, while features associated with large phase differences may be reduced, similar to interference phenomena caused by coherent light.

Specifically, when two feature blocks are aggregated with each other, their aggregation results are shown in Figure 10. Among them, the light dotted line and the dark solid line represent the original feature blocks, and the solid line represents the aggregation result. The phase difference between two feature blocks modulates their aggregation process.

In one implementation, the dot product result of qi and kj can be directly determined as the correlation degree. Since the correlation degree calculated above needs to be weighted with the V matrix, the elements in the first attention matrix can be mapped is a real number, obtain the second attention matrix, and obtain the output of the head according to the second attention matrix and the V matrix calculated in the head.

In one implementation, the dot multiplication result can be divided by a constant, and then the softmax operation is performed. Since the softmax operation needs to be performed in the real number domain, the result of the operation between the Q matrix and the K matrix (that is, the object of the softmax operation ) is a complex number. Therefore, in the embodiment of the present application, the result of the operation between the Q matrix and the K matrix (the first attention matrix) can be mapped to the real number domain.

In one implementation, the elements in the first attention matrix include real parts and imaginary parts. When mapping the elements in the first attention matrix to real numbers, the elements in the first attention matrix can be The numerical value of the real part and the numerical value of the imaginary part are fused to obtain a fusion result, and the fusion result is a real number.

For example, the fusion may include: a summation operation; or, finding the modulus length of a complex number.

In one implementation, the calculated complex-valued attention matrix (or dot product result is divided by a constant, and then softmax operation) is:

The above formula can be expanded into the form of real part and imaginary part respectively to obtain the attention matrix in the complex domain.

Based on the complex-valued attention matrix, real number domain features are determined. For example, through an additional linear transformation layer, the real part and the imaginary part are summed, and finally the real-valued attention matrix A is obtained:

Multiply A by value V to get the model output Y:
Y=AX, V=W _v X;

The output Y will be fed to the next layer of the model.

Experiments are conducted on large-scale data sets to conduct empirical research on the methods proposed in the embodiments of this application. Table 1 can illustrate that using quantum feature blocks can achieve significant accuracy improvements.

Table 1 Experimental results on ImageNet data set

Referring to Figure 11, Figure 11 is a schematic diagram of the system architecture or scenario of the application of the present invention. The present application can be embedded in different models and can be used in different tasks such as image understanding and natural language processing, and is actually deployed on mobile phones, watches, On various devices such as self-driving cars.

The embodiment of the present application provides a data processing method applied to the attention head in the neural network based on the attention mechanism. The method includes: performing multiple linear transformations on the feature blocks input to the head, Obtain the first amplitude, the second phase value, the second amplitude and the second phase value respectively; according to the first numerical value and the first phase value, obtain the Q matrix; the first amplitude is used as Q The amplitude of the elements in the matrix, the first phase value is used as the phase of the element in the Q matrix; according to the second amplitude and the second phase value, the K matrix is obtained; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the phase of the element in the K matrix; according to the V matrix calculated in the Q matrix, the K matrix and the head, Get the output of the head.

In the existing implementation, the transformation matrix Q and transformation matrix K are used to linearly transform the feature block token input into the head, and the Q matrix and K matrix are obtained respectively. The elements in the Q matrix and K matrix are all real numbers. For Features have limited expressive power. In this application, the amplitude and phase of the elements in the Q matrix and the amplitude and phase of the elements in the K matrix can be generated respectively for the feature block token input into the head. In this way, both the Q matrix and the K matrix can be parameterized into complex forms, thereby increasing the expressive ability of features and better modeling the relationship between different feature blocks.

Referring to Figure 12, Figure 12 is a flow diagram of a data processing method. As shown in Figure 12, the data processing method provided by the embodiment of the present application may include:

1201. By performing multiple linear transformations on the first feature block in the feature map, the first amplitude value and the first phase value are obtained respectively.

For the feature map that needs to be processed by the convolution layer, each feature block in the feature map (the embodiment of this application uses the first Taking the feature block as an example) perform multiple linear transformations to obtain the first amplitude and the first phase value respectively. The first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used to As the phase of the elements in the second feature block.

In a possible implementation, multiple linear transformations can be performed on the first feature block in the feature map through a linear programming layer; the linear programming layer includes a first weight and a second weight, and the first weight and the The second weight is a trainable parameter; wherein the first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude; the second weight is used to linearly transform the feature The first feature block in the figure is linearly transformed to obtain the first phase value.

1202. Obtain a second feature block according to the first amplitude value and the first phase value; the first amplitude value is used as the amplitude value of the element in the second feature block, and the first phase value is used for As the phase of the elements in the second feature block.

In one possible implementation, a convolutional neural network processes input data by stacking multiple convolutional layers to finally get the model input. Taking the convolutional neural network used for image recognition as an example, the middle layer of the network is a feature map of size H*W*C, where H, W, and C are the height, width, and number of channels of the model respectively. The characteristics of each position (1x1xC) can be regarded as a feature block, and each feature block can be expressed in the form of a wave function, that is, a quantum feature block. A basic unit in the convolutional neural network based on quantum feature blocks is shown in Figure 13:

First, two linear transformation layers W1 and W2 are used to represent the input feature X into a quantum feature block.

Different from existing methods, each feature block uses two linear transformation layers W1 and W2 to generate amplitude and phase respectively, so that the generated optimized feature blocks can express richer information.

1203. Map the elements in the second feature block to real numbers to obtain a third feature block.

In a possible implementation, the values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers can be fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion may include: a concatenation operation (concat).

For example, Euler's formula can be used to expand the complex-valued feature (that is, the second feature block) to obtain:

Based on complex-valued features, the real number domain features are determined. For example, splicing the real part and imaginary part of the complex-valued feature to obtain the real-number domain feature Z:
Z＝cat[W ₁ X⊙cosW ₂ X,W ₁ X⊙sinW ₂ X];

1204. Input the third feature block into the convolution layer.

Deploy convolution on feature Z to get model output Y:
Y=Conv(Z);

This application represents feature blocks as quantum feature blocks, and the model performs calculations in the complex number domain, which can achieve more accurate reasoning. In the final output layer of the network, the features are transformed into the real number domain through feature transformation to obtain practical meaningful model output.

This application can also achieve better performance by combining quantum feature blocks with convolutional neural networks. Their results on the target detection data set COCO are shown in Table 2:

Table 2 Experimental results of the target detection task on the COCO data set

This application provides a data processing method. The method includes: performing multiple linear transformations on the first feature block in the feature map to obtain the first amplitude and the first phase value respectively; according to the first amplitude and the first phase value to obtain a second feature block; the first amplitude value is used as the amplitude of the element in the second feature block, and the first phase value is used as the element in the second feature block phase; map the elements in the second feature block to real numbers to obtain the third feature block; input the third feature block into the convolution layer. This application introduces quantum feature blocks into the convolution operation. By representing the feature blocks as quantum feature blocks, the model can be calculated in the complex number domain and more accurate reasoning can be achieved. In the final output layer of the network, the features are transformed into the real number domain through feature transformation to obtain practical meaningful model output.

On the basis of the embodiments corresponding to Figures 1 to 13, in order to better implement the above solutions of the embodiments of the present application, relevant equipment for implementing the above solutions is also provided below. Specifically referring to Figure 14, Figure 14 is a schematic structural diagram of a data processing device 1400 provided by an embodiment of the present application. The data processing device 1400 may be a terminal device or a server. The data processing device 1400 includes:

The linear transformation module 1401 is used to perform multiple linear transformations on the feature blocks input into the head to obtain the first amplitude, the second phase value, the second amplitude and the second phase value respectively;

For a specific description of the linear transformation module 1401, reference may be made to step 601 in the above embodiment, which will not be described again here.

The attention calculation module 1402 is used to obtain the Q matrix according to the first numerical value and the first phase value; the first amplitude value is used as the amplitude of the element in the Q matrix, and the first phase value is used Yu is the phase of the element in the Q matrix;

According to the second amplitude and the second phase value, a K matrix is obtained; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the element in the K matrix. the phase of an element;

According to the Q matrix, the K matrix and the V matrix calculated in the head, the output of the head is obtained.

For a specific description of the attention calculation module 1402, reference may be made to step 602, step 603, and step 604 in the above embodiment, which will not be described again here.

In a possible implementation, the attention calculation module is specifically used to:

Perform correlation calculation on the Q matrix and the K matrix to obtain a first attention matrix, where the elements in the first attention matrix are complex numbers;

Map the elements in the first attention matrix to real numbers to obtain a second attention matrix;

According to the second attention matrix and the V matrix calculated in the head, the output of the head is obtained.

In a possible implementation, the elements in the first attention matrix include real parts and imaginary parts, and the attention calculation module is specifically used to:

The values of the real part and the value of the imaginary part of the elements in the first attention matrix are fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion includes:

sum operation; or,

Find the modular length of a complex number.

In a possible implementation, the linear transformation module is specifically used for:

Through the linear planning layer, multiple linear transformations are performed on the feature blocks input into the head; the linear planning layer includes a first weight, a second weight, a third weight and a fourth weight. The first weight, the The second weight, the third weight and the fourth weight are trainable parameters; wherein,

The first weight is used to linearly transform the feature block to obtain the first amplitude;

The second weight is used to linearly transform the feature block to obtain the first phase value;

The third weight is used to linearly transform the feature block to obtain the second amplitude;

The fourth weight is used to linearly transform the feature block to obtain the second phase value.

In a possible implementation, the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data.

Referring to Figure 15, Figure 15 is a schematic structural diagram of a data processing device 1500 provided by an embodiment of the present application. The data processing device 1500 may be a terminal device or a server. The data processing device 1500 includes:

The linear transformation module 1501 is used to perform multiple linear transformations on the first feature block in the feature map to obtain the first amplitude and the first phase value respectively;

According to the first amplitude and the first phase value, a second feature block is obtained; the first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used as the The phase of the elements in the second feature block;

Map elements in the second feature block to real numbers to obtain a third feature block;

For detailed description of the linear transformation module 1501, reference may be made to step 1201, step 1202, and step 1203 in the above embodiment, which will not be described again here.

The convolution module 1502 is used to input the third feature block into the convolution layer.

For a detailed description of the convolution module 1502, reference may be made to step 1204 in the above embodiment, which will not be described again here.

In a possible implementation, the linear transformation module is specifically used for:

The values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers are fused to obtain a fusion result, and the fusion result is a real number.

In a possible implementation, the fusion includes:

Concatenation operation (concat).

In a possible implementation, the linear transformation module is specifically used for:

Through the linear programming layer, multiple linear transformations are performed on the first feature block in the feature map; the linear programming layer includes a first weight and a second weight, and the first weight and the second weight are trainable parameters. ;in,

The first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude;

The second weight is used to linearly transform the first feature block in the feature map to obtain the first phase value.

Next, an execution device provided by an embodiment of the present application is introduced. Please refer to Figure 16. Figure 16 is a schematic structural diagram of an execution device provided by an embodiment of the present application. The execution device 1600 can be embodied as a virtual reality VR device, a mobile phone, or a virtual reality device. Computers, tablets, laptops, smart wearable devices, monitoring data processing equipment or servers, etc. are not limited here. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603 and a memory 1604 (the number of processors 1603 in the execution device 1600 can be one or more, one processor is taken as an example in Figure 16) , wherein the processor 1603 may include an application processor 16031 and a communication processor 16032. In some embodiments of the present application, the receiver 1601, the transmitter 1602, the processor 1603, and the memory 1604 may be connected by a bus or other means.

Memory 1604 may include read-only memory and random access memory and provides instructions and data to processor 1603 . A portion of memory 1604 may also include non-volatile random access memory (NVRAM). The memory 1604 stores processor and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operating instructions may include various operating instructions for implementing various operations.

The processor 1603 controls the execution of operations of the device. In specific applications, various components of the execution device are coupled together through a bus system. In addition to the data bus, the bus system may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are called bus systems in the figure.

The methods disclosed in the above embodiments of the present application can be applied to the processor 1603 or implemented by the processor 1603. The processor 1603 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 1603 . The above-mentioned processor 1603 can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field programmable Gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1603 can implement or execute each method, step and logical block diagram disclosed in the embodiment of this application. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 1604. The processor 1603 reads the information in the memory 1604 and completes the steps of the above method in combination with its hardware.

The receiver 1601 may be configured to receive input numeric or character information and generate signal inputs related to performing relevant settings and functional controls of the device. The transmitter 1602 can be used to output numeric or character information through the first interface; the transmitter 1602 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1602 can also include a display device such as a display screen .

In the embodiment of the present application, in one case, the processor 1603 is used to execute the data processing method executed by the device in the corresponding embodiment of FIG. 6 and FIG. 12 .

The embodiment of the present application also provides a training device. Please refer to Figure 17. Figure 17 is a schematic structural diagram of the training device provided by the embodiment of the present application. Specifically, the training device 1700 is implemented by one or more servers. The training device 1700 There may be relatively large differences due to different configurations or performance, which may include one or more central processing units (CPU) 1717 (for example, one or more processors) and memory 1732, one or one The above storage medium 1730 (such as one or more mass storage devices) stores application programs 1742 or data 1744. Among them, the memory 1732 and the storage medium 1730 may be short-term storage or persistent storage. The program stored in the storage medium 1730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1717 may be configured to communicate with the storage medium 1730 and execute a series of instruction operations in the storage medium 1730 on the training device 1700 .

The training device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input and output interfaces 1758; or, one or more operating systems 1741, such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.

In the embodiment of the present application, the central processor 1717 is used to execute the data processing method executed by the data processing device in the corresponding embodiment of FIG. 6 and FIG. 12 .

An embodiment of the present application also provides a computer program product that, when run on a computer, causes the computer to perform the steps performed by the foregoing execution device, or causes the computer to perform the steps performed by the foregoing training device.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a program for performing signal processing. When the program is run on a computer, it causes the computer to perform the steps performed by the aforementioned execution device. , or, causing the computer to perform the steps performed by the aforementioned training device.

The execution device, training device or terminal device provided by the embodiment of the present application may specifically be a chip. The chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor. The communication unit may be, for example, an input/output interface. Pins or circuits, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.

Specifically, please refer to Figure 18. Figure 18 is a structural schematic diagram of a chip provided by an embodiment of the present application. The chip can be represented as a neural network processor NPU 1800. The NPU 1800 serves as a co-processor and is mounted to the main CPU (Host). CPU), tasks are allocated by the Host CPU. The core part of the NPU is the arithmetic circuit 1803. The arithmetic circuit 1803 is controlled by the controller 1804 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the computing circuit 1803 includes multiple processing units (Process Engine, PE). In some implementations, arithmetic circuit 1803 is a two-dimensional systolic array. The arithmetic circuit 1803 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1803 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit obtains the corresponding data of matrix B from the weight memory 1802 and caches it on each PE in the arithmetic circuit. The operation circuit takes matrix A data and matrix B from the input memory 1801 to perform matrix operations, and the partial result or final result of the matrix is stored in an accumulator (accumulator) 1808 .

The unified memory 1806 is used to store input data and output data. The weight data directly passes through the storage unit access controller (Direct Memory Access Controller, DMAC) 1805, and the DMAC is transferred to the weight memory 1802. Input data is also transferred to unified memory 1806 via DMAC.

BIU is the Bus Interface Unit, that is, the bus interface unit 1810, which is used for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1809.

The bus interface unit 1810 (Bus Interface Unit, BIU for short) is used to fetch the memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1806 or the weight data to the weight memory 1802 or the input data to the input memory 1801 .

The vector calculation unit 1807 includes multiple arithmetic processing units, and if necessary, further processes the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. Mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

In some implementations, vector calculation unit 1807 can store the processed output vectors to unified memory 1806 . For example, the vector calculation unit 1807 can apply a linear function; or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation on the feature plane extracted by the convolution layer, or a vector of accumulated values, to generate an activation value. In some implementations, vector calculation unit 1807 generates normalized values, pixel-wise summed values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 1803, such as for use in a subsequent layer in a neural network.

The instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;

The unified memory 1806, the input memory 1801, the weight memory 1802 and the fetch memory 1809 are all On-Chip memories. External memory is private to the NPU hardware architecture.

The processor mentioned in any of the above places can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

In addition, it should be noted that the device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physically separate. The physical unit can be located in one place, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between modules indicates that there are communication connections between them, which can be specifically implemented as one or more communication buses or signal lines.

Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, Special components, etc. to achieve. In general, all functions performed by computer programs can be easily implemented with corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special-purpose circuits. circuit etc. However, for this application more often the software program implementation is Better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a readable storage medium, such as a computer floppy disk. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to cause a computer device (which can be a personal computer, training device, or network device, etc.) to execute the steps described in various embodiments of this application. method.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transferred from a website, computer, training device, or data The center transmits to another website site, computer, training equipment or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a training device or a data center integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state disk (Solid State Disk, SSD)), etc.

Claims (23)

A data processing method, characterized in that it is applied to the attention head in a neural network based on an attention mechanism, and the method includes: By performing multiple linear transformations on the feature blocks input to the head, the first amplitude, the second phase value, the second amplitude and the second phase value are obtained respectively; According to the first numerical value and the first phase value, a Q matrix is obtained; the first amplitude value is used as the amplitude of the element in the Q matrix, and the first phase value is used as the element in the Q matrix phase; According to the second amplitude and the second phase value, a K matrix is obtained; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the element in the K matrix. the phase of an element; According to the Q matrix, the K matrix and the V matrix calculated in the head, the output of the head is obtained. The method of claim 1, wherein obtaining the output of the head based on the Q matrix, the K matrix and the V matrix calculated in the head includes: Perform correlation calculation on the Q matrix and the K matrix to obtain a first attention matrix, where the elements in the first attention matrix are complex numbers; Map the elements in the first attention matrix to real numbers to obtain a second attention matrix; According to the second attention matrix and the V matrix calculated in the head, the output of the head is obtained. The method according to claim 2, wherein the elements in the first attention matrix include real parts and imaginary parts, and mapping the elements in the first attention matrix to real numbers includes: The values of the real part and the value of the imaginary part of the elements in the first attention matrix are fused to obtain a fusion result, and the fusion result is a real number. The method according to claim 3, characterized in that the fusion includes: sum operation; or, Find the modular length of a complex number. The method according to any one of claims 1 to 4, characterized in that by performing multiple linear transformations on the elements of the feature block input to the head, the first amplitude, the first phase value, The second amplitude value and the second phase value include: Through the linear planning layer, multiple linear transformations are performed on the feature blocks input into the head; the linear planning layer includes a first weight, a second weight, a third weight and a fourth weight. The first weight, the The second weight, the third weight and the fourth weight are trainable parameters; wherein, The first weight is used to linearly transform the feature block to obtain the first amplitude; The second weight is used to linearly transform the feature block to obtain the first phase value; The third weight is used to linearly transform the feature block to obtain the second amplitude; The fourth weight is used to linearly transform the feature block to obtain the second phase value. The method according to any one of claims 1 to 5, characterized in that the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data. A data processing method, characterized in that the method includes: By performing multiple linear transformations on the first feature block in the feature map, the first amplitude and first phase values are obtained respectively; According to the first amplitude and the first phase value, a second feature block is obtained; the first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used as the The phase of the elements in the second feature block; Map elements in the second feature block to real numbers to obtain a third feature block; The third feature block is input into the convolutional layer. The method of claim 7, wherein mapping elements in the second feature block to real numbers includes: The values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers are fused to obtain a fusion result, and the fusion result is a real number. The method according to claim 8, characterized in that the fusion includes: Concatenation operation (concat). The method according to any one of claims 7 to 9, characterized in that the first amplitude value and the first phase value are respectively obtained by performing multiple linear transformations on the first feature block in the feature map, including: Through the linear programming layer, multiple linear transformations are performed on the first feature block in the feature map; the linear programming layer includes a first weight and a second weight, and the first weight and the second weight are trainable parameters. ;in, The first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude; The second weight is used to linearly transform the first feature block in the feature map to obtain the first phase value. A data processing device, characterized in that it is applied to the attention head in a neural network based on an attention mechanism, and the device includes: A linear transformation module, used to perform multiple linear transformations on the feature blocks input into the head to obtain the first amplitude, the second phase value, the second amplitude and the second phase value respectively; Attention calculation module, used to obtain the Q matrix according to the first numerical value and the first phase value; the first amplitude value is used as the amplitude value of the element in the Q matrix, and the first phase value is used to As the phase of the elements in the Q matrix; According to the second amplitude and the second phase value, a K matrix is obtained; the second amplitude is used as the amplitude of the element in the K matrix, and the second phase value is used as the element in the K matrix. the phase of an element; According to the Q matrix, the K matrix and the V matrix calculated in the head, the output of the head is obtained. The device according to claim 11, characterized in that the attention calculation module is specifically used to: Perform correlation calculation on the Q matrix and the K matrix to obtain a first attention matrix, where the elements in the first attention matrix are complex numbers; Map the elements in the first attention matrix to real numbers to obtain a second attention matrix; According to the second attention matrix and the V matrix calculated in the head, the output of the head is obtained. The device according to claim 12, wherein the elements in the first attention matrix include real parts and imaginary parts, and the attention calculation module is specifically used for: The values of the real part and the value of the imaginary part of the elements in the first attention matrix are fused to obtain a fusion result, and the fusion result is a real number. The device according to claim 13, wherein the fusion includes: sum operation; or, Find the modular length of a complex number. The device according to any one of claims 11 to 14, characterized in that the linear transformation module is specifically used for: Through the linear planning layer, multiple linear transformations are performed on the feature blocks input into the head; the linear planning layer includes a first weight, a second weight, a third weight and a fourth weight. The first weight, the The second weight, the third weight and the fourth weight are trainable parameters; wherein, The first weight is used to linearly transform the feature block to obtain the first amplitude; The second weight is used to linearly transform the feature block to obtain the first phase value; The third weight is used to linearly transform the feature block to obtain the second amplitude; The fourth weight is used to linearly transform the feature block to obtain the second phase value. The device according to any one of claims 11 to 15, wherein the feature block is information associated with a slice of a piece of data, and the piece of data is audio data, video data, image data or context data. A data processing device, characterized in that the device includes: A linear transformation module, used to perform multiple linear transformations on the first feature block in the feature map to obtain the first amplitude and the first phase value respectively; According to the first amplitude and the first phase value, a second feature block is obtained; the first amplitude is used as the amplitude of the element in the second feature block, and the first phase value is used as the The phase of the elements in the second feature block; Map elements in the second feature block to real numbers to obtain a third feature block; The convolution module is used to input the third feature block into the convolution layer. The device according to claim 17, characterized in that the linear transformation module is specifically used for: The values of the real part and the value of the imaginary part when the elements in the second feature block are expressed as complex numbers are fused to obtain a fusion result, and the fusion result is a real number. The device according to claim 18, wherein the fusion includes: Concatenation operation (concat). The device according to any one of claims 17 to 19, characterized in that the linear transformation module is specifically used for: Through the linear programming layer, multiple linear transformations are performed on the first feature block in the feature map; the linear programming layer includes a first weight and a second weight, and the first weight and the second weight are trainable parameters. ;in, The first weight is used to linearly transform the first feature block in the feature map to obtain the first amplitude; The second weight is used to linearly transform the first feature block in the feature map to obtain the first phase value. A data processing device, characterized in that the device includes a memory and a processor; the memory stores code, and the processor is configured to obtain the code and execute as described in any one of claims 1 to 10 Methods. A computer storage medium, characterized in that the computer storage medium stores one or more instructions, which when executed by one or more computers cause the one or more computers to implement any of claims 1 to 10. The method described in 1. A computer program product, characterized in that it includes computer-readable instructions that, when run on a computer device, cause the computer device to perform the method according to any one of claims 1 to 10.