TW202230221A - Neural architecture scaling for hardware accelerators - Google Patents

Abstract

Methods, systems, and apparatus, including computer-readable media, for scaling neural network architectures on hardware accelerators. A method includes receiving training data and information specifying target computing resources, and performing using the training data, a neural architecture search over a search space to identify an architecture for a base neural network. A plurality of scaling parameter values for scaling the base neural network can be identified, which can include repeatedly selecting a plurality of candidate scaling parameter values, and determining a measure of performance for the base neural network scaled according to the plurality of candidate scaling parameter values, in accordance with a plurality of second objectives including a latency objective. An architecture for a scaled neural network can be determined using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

Description

Neural Architecture Scaling for Hardware Accelerators

A neural network is a machine learning model that includes one or more layers of nonlinear operations to predict an output for a received input. In addition to an input layer and an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer can be input to another hidden or output layer of the neural network. Each layer of the neural network can generate a respective output from a received input according to the value of one or more model parameters of the layer. Model parameters may be weights or biases determined by a training algorithm to cause the neural network to produce accurate outputs.

A system implemented in accordance with aspects of the present invention can reduce the latency of a neural network architecture by searching for candidate neural network architectures together according to each candidate's computational requirements (eg, FLOPS), operational intensity, and execution efficiency. Computational requirements, operational intensity, and execution efficiency were found to be one of the root causes of a neural network's latency on target computational resources, while non-computational requirements alone affect latency (including inference latency), as described herein. Aspects of the present invention provide techniques for performing neural architecture search and scaling, such as by delay-aware compound scaling and by augmentation based on this observed relationship between delay and computation, operational intensity, and execution efficiency from which to search candidate neurons cyberspace.

Furthermore, the system can perform compound scaling to scale multiple parameters of a neural network consistently and according to multiple objectives, which can result in advantages over methods in which a single objective is considered or in which the scaling parameters of a neural network are searched separately Improved performance of the scaled neural network. Delay-aware compound scaling can be used to quickly build a series of neural network architectures, scaled according to different values from an initial scaled neural network architecture, and can be adapted to different use cases.

According to aspects of the invention, a computer-implemented method includes a method for determining an architecture of a neural network comprising: receiving, by one or more processors, training data corresponding to a neural network task and specifying target computing resource information; identifying an architecture of a basic neural network by the one or more processors using the training data and performing a neural architecture search on a search space according to a plurality of first targets; by the One or more processors identify a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network. The identifying may include repeatedly performing the steps of: selecting a plurality of candidate scaling parameter values; and determining an efficacy measure of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein according to a plurality of second ones including a delay target target to determine the efficacy measure. The method may include generating, by the one or more processors, an architecture of a scaled neural network using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

The foregoing and other embodiments may each include one or more of the following features, alone or in combination.

The plurality of first targets used to perform the neural architecture search may be the same as the plurality of second targets used to identify the plurality of scaling parameter values.

The plurality of first targets and the plurality of second targets may include an accuracy target corresponding to the accuracy of the output of the base neural network.

When the base neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources, the efficiency metric may be at least partially associated with a relationship between the base neural network receiving an input and generating an output Latency metric corresponds.

When the basic neural network is deployed on the target computing resources, the delay target may correspond to a minimum delay between the basic neural network receiving an input and generating an output.

The search space may include candidate neural network layers, each candidate neural network layer configured to perform one or more respective operations. The search space may include candidate neural network layers that include different respective excitation functions.

The architecture of the basic neural network may include a plurality of components, each component having a respective plurality of neural network layers. The search space may include a plurality of candidate components of the candidate neural network layer, including: a first component of the candidate network layer, which includes a first excitation function; and a second component of the candidate network layer, which includes different a second excitation function in the first excitation function.

The information specifying the target computing resources may specify one or more hardware accelerators; and wherein the method further includes executing the scaled neural network on the one or more hardware accelerators to perform the neural network task.

The target computing resources may include a first target computing resource, the plurality of scaling parameter values are a plurality of first scaling parameter values, and the method may further include: receiving, by the one or more processors, a designation different from the information of a second target computing resource of a first target computing resource; and identifying a plurality of second scaling parameter values for scaling the basic neural network according to the information specifying the second target computing resources, wherein the plurality of The second scaling parameter value is different from the plurality of first scaling parameter values.

The plurality of scaling parameter values are a plurality of first scaling parameter values, and wherein the method further comprises: generating a scaled neural network architecture from the base neural network architecture scaled using the plurality of second scaling parameter values, wherein according to the A plurality of first scaling parameter values and one or more composite coefficients that uniformly modify the value of each of the first scaling parameter values generate the second scaling parameter values.

The basic neural network can be a convolutional neural network, and wherein the plurality of scaling parameters can include a depth of the basic neural network, a width of the basic neural network, and an input resolution of the basic neural network one or more.

According to another aspect, a method for determining an architecture of a neural network includes: receiving, by one or more processors, information specifying a target computing resource; receiving, by the one or more processors, specifying a basic Data of an architecture of a neural network; identified by the one or more processors for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network A plurality of scaling parameter values. The identifying may include repeatedly performing the steps of: selecting a plurality of candidate scaling parameter values; and determining an efficacy measure of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein according to a plurality of targets including a delay target Determining the performance measure; and generating, by the one or more processors, an architecture of a scaled neural network using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

The plurality of targets can be a plurality of second targets; and receiving the data specifying the architecture of the base neural network can include: receiving, by one or more processors, training data corresponding to a neural network task; and The architecture of the base neural network is identified by the one or more processors using the training data and performing a neural architecture search on a search space according to a plurality of first targets.

Other implementations include computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Cross-references to related applications

This application claims the benefit of US Patent Application Serial No. 63/137,926, filed January 15, 2021, under 35 U.S.C. §119(e), the disclosure of which is incorporated herein by reference. Overview:

The techniques described in this specification generally relate to scaling neural networks that execute on different target computing resources, such as different hardware accelerators. Neural networks may be scaled according to a number of different performance goals, which may include minimizing processing time (referred to herein as latency) and maximizing neural network when scaling on target computing resources to execute. A separate goal of network precision.

In general, a neural architecture search (NAS) system can be deployed to select a neural network architecture from a given search space of one of candidate architectures according to one or more objectives. A common target is the accuracy of neural networks, and generally, implementing a system of NAS techniques will favor networks that result in higher accuracy after training, rather than networks with lower accuracy. Immediately after a NAS selects a base neural network, the base neural network may be scaled according to one or more scaling parameters. Scaling may include searching for one or more scaling parameter values for scaling the base neural network, eg, by searching for scaling parameters in a coefficient search space of a number. Scaling can include increasing or decreasing the number of layers or the size of layers that a neural network has to efficiently utilize the computing and/or memory resources available to deploy the neural network.

A common concept associated with neural architecture search and scaling is that a network processes the computational requirements (eg, measured in floating point operations per second (FLOPS)) required by a neural network to process an input and sends an input The delay between going to the network and receiving an output is proportional. In other words, a neural network with a low operation (low FLOPS) requirement is believed to produce output faster than a network with a higher operation (high FLOPS) requirement because fewer operations are performed overall. Therefore, many NAS systems choose neural networks with a low computational requirement. However, since other characteristics of neural networks, such as operational intensity, parallelism, and execution efficiency, can affect the overall latency of a neural network, it has been determined that the computational requirements are not proportional to the latency.

The techniques described herein provide delay-aware composite scaling (LACS) and augmentation of the candidate neural network search space from which a neural network is selected. In the context of augmenting a search space, "hardware accelerator friendly" operations and architectures can be included in the search space, where such additions can result in higher operational intensity, execution efficiency, and suitability for deployment in various types of hardware Parallelism on bulk accelerators. Such operations may include space-to-depth operations, space-to-batch operations, fusion convolutional structures, and component-wise search for excitation functions.

Delay-aware composite scaling of neural networks can improve the scaling of neural networks relative to conventional methods that are not optimized for delay. Instead, LACS can be used to identify scaled parameter values for scaled neural networks that are accurate and operate with low latency on target computing resources.

The technique further provides multi-target scaling that is shared for searching a neural architecture target using NAS or similar techniques. Scaled neural networks can be identified based on the same targets used when searching for the base neural network. Thus, a scaled neural network can optimize performance in two phases (basic architecture search and scaling), rather than treating each phase as a task with separate goals.

LACS can be integrated with existing NAS systems, at least because the same goals for both search and scaling can be used to create an end-to-end system for determining a scaled neural network architecture. Furthermore, a series of scaled neural network architectures, where the neural network architectures are searched but not scaled for deployment on target computing resources, can be identified faster than unscaled search methods.

The techniques described herein may provide improved neural networks compared to neural networks identified in conventional search and scaling methods that do not use LACS. Additionally, a family of neural networks with different tradeoffs between goals such as model accuracy and inference latency can be quickly generated for various use cases. Furthermore, this technique may provide faster identification of neural networks used to perform a particular task, but the identified neural networks may perform with improved accuracy over neural networks identified using other methods. This is at least because neural networks identified as a result of searching and scaling as described herein can take into account characteristics that can affect latency, such as operational intensity and execution efficiency, rather than just the computational requirements of a network. In this way, one of the identified neural networks can perform inference faster without sacrificing the accuracy of the network.

This technique may further provide a generally applicable framework for rapidly migrating existing neural networks to an improved computing resource environment. For example, LACS and NAS as described herein may be applied when the execution of an existing neural network selected for a data center with specific hardware is migrated to a data center using a different hardware. In this regard, a series of neural networks can be quickly identified to perform the tasks of existing neural networks and deployed on hardware in new data centers. This application may be particularly useful in rapidly evolving fields that require state-of-the-art hardware to perform efficiently, such as networks performing tasks in computer vision or other image processing tasks.

1 is a block diagram illustrating a series 103 of scaled neural network architectures 104A-N deployed in a data center 115 housing a hardware accelerator 116 on which the deployed neural network will execute. The hardware accelerator 116 may be any type of processor, such as a CPU, GPU, FGPA or ASIC, such as a TPU. According to aspects of the invention, the series 103 of scaled neural network architectures can be generated from a base neural network architecture 101 .

An architecture of a neural network refers to the properties that define the neural network. For example, the architecture may include the characteristics of the various neural network layers of the network, how the layers process input, how the layers interact with each other, and so on. For example, a convolutional neural network (ConvNet) architecture may define a discrete convolutional layer that receives input image data, followed by a pooling layer, which is then followed by generating an output based on a neural network task (eg, for the input image The content of the data is classified) a fully connected layer. The architecture of a neural network can also define the types of operations performed within each layer. For example, the architecture of a ConvNet can define the use of ReLU excitation functions in the fully connected layers of the network.

The basic neural network architecture 101 can be identified from a set of objectives and using NAS. The base neural network architecture 101 can be identified from the search space from one of the candidate neural network architectures according to the set of objectives and using NAS. As described in more detail herein, the search space of candidate neural network architectures can be augmented to include different network components, operations, and layers, from which a base network that satisfies a goal can be identified.

The set of objectives used to identify the base neural network architecture 101 can also be applied to identify scaling parameter values for each of the neural networks 104A-N in the series 103 . The base neural network architecture 101 and the scaled neural network architectures 104A-N can be characterized by several parameters, which are scaled to different degrees in the scaled neural network architectures 104A-N. In Figure 1, neural networks 101, 104A are shown using three scaling parameters: D indicates the number of layers in the neural network; W indicates the width or number of neurons within a neural network layer; The size of the input processed by the way at a given layer.

As described in more detail herein, a system configured to perform LACS may search a coefficient search space 108 to identify sets of scaling parameter values. Each scaling parameter value is a coefficient in the coefficient search space, which can be, for example, a set of positive real numbers. Each candidate network 107A to N is scaled from the base neural network 101 according to the candidate coefficient values identified as part of a search in the coefficient search space 108 . The system may apply any of a variety of different search techniques to identify candidate coefficients, such as a Pareto leading edge search or a trellis search. For each candidate network 107A-N, the system may evaluate a measure of the effectiveness of the candidate network in performing a neural network task. The efficacy metric may be based on a number of objectives, including a latency objective for measuring the delay between receiving an input for a candidate network and producing a corresponding output as part of performing a neural network task.

After performing a search for scaled parameter values on the coefficient search space 108 , the system may receive a scaled neural network architecture 109 . The scaled neural network architecture 109 scales from the base neural network 101, where the scaling parameter values result in the highest performance metric for candidate networks 107A-N identified during a search in the coefficient search space.

From the scaled neural network architecture 109, the system can generate the series 103 of scaled neural network architectures 104A-N. The series 103 can be generated by scaling the scaled neural network architecture 109 according to different values. The various scaling parameter values of the scaled neural network architecture 109 can be scaled consistently to generate the different scaled neural network architectures in the series 103 . For example, each scaling parameter value of the scaled neural network architecture 109 may be scaled by increasing each scaling parameter value by a factor of 2. The scaled neural network architecture 109 may be scaled for different values or "composite coefficients" of each scaling parameter value that are consistently applied to the scaled neural network architecture 109 . In some implementations, the scaled neural network architecture 109 is scaled in other ways, eg, by separately scaling each scaling parameter value to produce one of the scaled neural network architectures in the series 103 .

By scaling the scaled neural network architecture 109 according to different values, different neural network architectures can be quickly generated to perform a task according to a variety of different use cases. Different use cases may be specified for identifying different tradeoffs among the multiple goals of the scaled neural network architecture 109 . For example, a scaled neural network architecture can be identified as meeting a higher accuracy threshold at the expense of higher latency during execution. Another scaled neural network architecture may be identified as meeting a lower accuracy threshold, but may execute on hardware accelerator 116 with lower latency. Another scaled neural network architecture that balances the tradeoff between accuracy and latency on the hardware accelerator 116 can be identified.

As an example, in order to perform a computer vision task (such as object recognition), a neural network architecture may be required to generate output in real-time or near real-time, as an application for continuously receiving video or image data and for identifying the Part of a task for an object of a specific class. In this exemplary task, the tolerance to accuracy may be lower, so a scaled neural network architecture that scales with an appropriate trade-off for lower latency and lower accuracy may be deployed to perform the task.

As another example, a neural network architecture may be tasked with classifying objects in a scene received from one of image or video data. In this example, if latency in performing this exemplary task is not considered as important as accurately performing the task, then a scaled neural network with higher accuracy at the expense of latency can be deployed. In other instances, scaled neural network architectures that balance tradeoffs between accuracy, latency, and other goals may be deployed where a particular tradeoff is not identified or desired when performing neural network tasks.

The scaled neural network architecture 104N is scaled using a different scaling parameter value than the scaling parameter value used to scale the scaled neural network architecture 109 to obtain the scaled neural network 104A, and may represent a different use case, eg, where Use cases where precision is expected rather than inference latency.

The LACS and NAS techniques described herein can generate series 103 for hardware accelerators 116 and receive additional training data and information specifying a number of different computing resources, such as different types of hardware accelerators. In addition to generating series 103 for hardware accelerators 116, the system can also search for a base neural network architecture and generate a series of scaled neural network architectures for different hardware accelerators. For example, given one GPU and one TPU, the system may generate separate model families optimized for the accuracy-latency tradeoff for the GPU and TPU, respectively. In some implementations, the system can generate multiple scaled series from the same basic neural network architecture. Exemplary method

2 is a flow diagram of an exemplary process 200 for generating a scaled neural network architecture for execution on a target computing resource. The exemplary process 200 may be executed on a system of one or more processors in one or more locations. For example, a Neural Architecture Search-Latency Aware Complex Scaling (NAS-LACS) system as described herein may execute the procedure 200 .

As shown in block 210, the system receives training data corresponding to a neural network task. A neural network task is a machine learning task that can be performed by a neural network. A scaled neural network can be configured to receive any type of data input to generate output for performing a neural network task. As an example, the output can be any type of scoring, classification, or recursive output based on the input. Accordingly, a neural network task may be a scoring, classification and/or recursive task for predicting some outputs given some inputs. These tasks may correspond to a variety of different applications for processing images, video, text, voice, or other types of data.

The training data received may be in any form suitable for training a neural network according to one of a variety of different learning techniques. Learning techniques for training a neural network can include supervised learning, unsupervised learning, and semi-supervised learning techniques. For example, training data may include training instances that may be received as input by a neural network. Training instances may be labeled with a known output corresponding to an output intended to be produced by a neural network that is properly trained to perform a particular neural network task. For example, if the neural network task is a classification task, the training example may be an image labeled with one or more classes that classify the objects depicted in the image.

As shown in block 220, the system receives information specifying a target computing resource. The target computing resource data may specify characteristics of the computing resource on which a neural network may be deployed at least in part. Computing resources may be housed in one or more data centers or other physical locations with any of a variety of different types of hardware devices. Exemplary types of hardware include central processing units (CPUs), graphics processing units (GPUs), edge or mobile computing devices, field programmable gate arrays (FGPAs), and various types of application specific circuits (ASICs).

Some devices may be configured for hardware acceleration, which may include devices configured to perform certain types of operations efficiently. Such hardware accelerators, which may include, for example, GPUs and tensor processing units (TPUs) may implement special features for hardware acceleration. Exemplary features for hardware acceleration may include configurations to perform operations typically associated with machine learning model execution, such as matrix multiplication. Such special features may also include, as examples, matrix multiply and accumulate units available in different types of GPUs, and matrix multiply units available in TPUs.

Target computing resource data may include data for one or more target sets of computing resources. A target set of computing resources may refer to a set of computing devices on which a neural network is desired to be deployed. Information specifying a target set of computing resources may refer to the type and number of hardware accelerators or other computing devices in the target set. The target set may contain devices of the same or different types. For example, a target set of computing resources may define the hardware characteristics and quantity of a particular type of hardware accelerator, including its processing power, throughput, and memory capacity. As described herein, a system can generate a series of scaled neural network architectures for each device specified in a target set of computing resources.

Additionally, target computing resource data may specify different target sets of computing resources, eg, reflecting different potential configurations of computing resources housed in a data center. From the training and target computing resource data, the system can generate a series of neural network architectures. Architectures can be generated from one of the basic neural networks identified by the system.

As shown in block 230, the system may perform a neural architecture search on a search space using the training data to identify an architecture of a basic neural network. The system may use any of a variety of NAS techniques, such as those based on reinforcement learning, evolutionary search, or differentiable search. In some implementations, the system can directly receive data specifying an architecture of an underlying neural network, eg, without receiving training data and performing NAS as described herein.

A search space refers to candidate neural networks or portions of candidate neural networks that can potentially be selected as part of a base neural network architecture. A portion of a candidate neural network architecture may refer to a component of the neural network. The architecture of a neural network can be defined in terms of a plurality of components of the neural network, each of which includes one or more neural network layers. The properties of a neural network layer can be defined in an architecture at the component level, which means that the architecture can define specific operations to be performed in a component such that each neural network in a component implements the same operations defined for the component. Components can also be defined in the architecture by the number of layers in the component.

As part of performing NAS, the system may repeatedly identify candidate neural networks, obtain performance metrics corresponding to multiple targets, and evaluate candidate neural networks, among others, based on their respective performance metrics. As part of obtaining performance metrics, such as measures of accuracy and latency of the candidate neural network, the system may use the received training data to train the candidate neural network. Once trained, the system can evaluate candidate neural network architectures to determine their performance metrics, and compare performance metrics against a current best candidate.

The system can repeat this search procedure by selecting candidate neural networks, training the networks, and comparing their performance metrics until a stopping criterion is reached. The stopping criterion may be a minimum predetermined performance threshold value satisfied by a current candidate network. Additionally or alternatively, the stopping criterion may be a maximum number of search iterations, or a maximum amount of time allotted to perform a search. The stopping criterion may be a condition in which the performance of the neural network converges, eg, the performance of a subsequent iteration is less than a threshold value that differs from the performance of the previous iteration.

In the context of optimizing a neural network for different performance metrics (eg, accuracy and latency), the stopping criterion may specify a range of thresholds that are pre-determined to be "best." For example, a threshold range of optimal delays may be derived from a theoretical or measured minimum delay threshold range achieved by the target computing resource. The theoretical or measured minimum latency may be based on physical characteristics of the computing resource, such as the minimum amount of time required for components of the computing resource to physically read and process incoming data. In some implementations, the latency is kept to an absolute minimum, eg, physically as close to zero latency as possible, and not based on a target latency measured or calculated from the target computing resources.

The system can be configured to use a machine learning model or other techniques to select the next candidate neural network architecture, wherein the selection can be based, at least in part, on different candidates that are more likely to perform well under the goals of a particular neural network task The learned properties of neural networks.

ACCURACY(m)係一候選神經網路m之精度之效能度量，且 LATENCY(m)係神經網路在於目標運算資源上產生一輸出時之延時。如本文中更詳細描述，精度及延時亦可為由系統作為根據目標運算資源之特性縮放基本神經網路架構之部分所使用之目標。 Target係一目標延時值，例如以毫秒為單位量測，且可經預先判定。值 w係用於對網路延時對候選神經網路之整體效能之影響進行加權之一可調諧參數。可根據運算資源之不同目標集來學習或調諧不同值 w。作為一實例， w可經設定為-0.09，從而反映適用於如TPU及GPU之運算資源之一整體較大因數，該等運算資源對延時變動之敏感度小於例如行動平台(其中 w可經設定為一較小值，諸如-0.07)。 In some instances, the system may use a multi-objective reward mechanism for identifying basic neural network architectures, as follows:

ACCURACY(m) is a performance measure of the accuracy of a candidate neural network m, and LATENCY(m) is the delay for the neural network to generate an output on the target computing resource. As described in more detail herein, accuracy and latency may also be goals used by the system as part of scaling the underlying neural network architecture according to the characteristics of the target computing resources. Target is a target delay value, measured in milliseconds, for example, and can be pre-determined. The value w is a tunable parameter used to weight the effect of network latency on the overall performance of the candidate neural network. Different values of w may be learned or tuned according to different target sets of computing resources. As an example, w may be set to -0.09, reflecting an overall larger factor applicable to computing resources such as TPUs and GPUs that are less sensitive to latency variations than, for example, mobile platforms (where w may be set is a small value, such as -0.07).

To measure the accuracy of a candidate neural network, the system can use the training set to train the candidate neural network to perform a neural network task. The system may divide the training data into a training set and a validation set, eg, according to an 80/20 split. For example, the system may apply a supervised learning technique to compute an error between the output produced by the candidate neural network and the true data label of a training instance processed by the network. The system may use any of a variety of loss or error functions appropriate for the type of task the neural network is trained for, such as cross-entropy loss for classification tasks, or mean squared error for recursive tasks. For example, a back-pass algorithm can be used to calculate the error gradient with respect to different weights of the candidate neural network, and the weights of the neural network can be updated. The system can be configured to train candidate neural networks until stopping criteria are met, such as a number of iterations for training, a maximum time period, convergence, or when a minimum accuracy threshold is met.

In addition to other performance metrics, the system can also generate performance metrics of the accuracy and latency of candidate neural network architectures on target computing resources, including (i) the operational strength of the selected basic neural network when deployed on target computing resources, and/or an execution efficiency of the candidate basic neural network on the target computing resource. In some implementations, in addition to accuracy and latency, performance metrics for candidate base neural networks are also based, at least in part, on their operational strength and/or execution efficiency.

Latency, operation intensity and execution efficiency can be defined as follows:

，其中 W係用於執行所量測之神經網路架構之運算量(例如，以FLOPS為單位)，且 C係藉由目標運算資源在神經網路架構上處理輸入而達成之運算速率(例如，以FLOPS/sec為單位)。 E係候選神經網路在目標運算資源上執行之執行效率，且 E可等於 C/C _ideal (且因此，藉由代數，

)。 C _ideal 係在用於執行所量測之神經網路架構之目標運算資源上達成之理想運算速率。 C _ideal 根據操作強度 I、目標運算資源之記憶體頻寬 b及 C _max 來定義， C _max 表示目標運算資源之峰值運算速率(例如，一GPU之峰值運算速率)。 In (3), LATENCY is defined as

, where W is the amount of computation (e.g., in FLOPS) used to execute the measured neural network architecture, and C is the computation rate achieved by the target computing resource processing input on the neural network architecture (e.g., , in FLOPS/sec). E is the execution efficiency of the candidate neural network on the target computing resource, and E can be equal to C/C _ideal (and thus, by algebra,

). C _ideal is the ideal computing rate achieved on the target computing resources used to execute the measured neural network architecture. C _ideal is defined according to the operation intensity I , the memory bandwidth b of the target computing resource, and C _max , where C _max represents the peak computing rate of the target computing resource (eg, the peak computing rate of a GPU).

，如(3)中展示)。 _Cmax and b are constants and correspond to the hardware characteristics of the target computing resource, such as its peak computing rate and memory bandwidth, respectively. Operational intensity I refers to a measure of the amount of data processed by the computing resources deploying a neural network, given the amount of computing W required for the neural network to execute, divided by the memory induced by the computing resources during the execution of the neural network Sports Q (

, as shown in (3)).

The ideal computational rate for executing the neural network on the target computational resource is Ixb when I<R , and _Cmax otherwise. R is the spine point, or the minimum operating intensity required by the neural network architecture to achieve peak computing rates on the target computing resources.

Collectively, (3) and the corresponding description show that the inference latency of a neural network once measured is based on operation intensity I , operation W and execution efficiency E , rather than operation W alone. In the context of hardware accelerators (specifically, hardware accelerators such as TPUs and GPUs commonly deployed in data centers), this relationship can be applied to use "accelerator friendly operations" to augment a candidate neural network during NAS The network search space, and how to refine how a basic neural network is selected and subsequently scaled by the system.

The system can search the base neural network architecture to improve the latency of a final neural network by simultaneously searching for candidate neural network architectures with improved operational strength, execution efficiency, and computing requirements, rather than searching separately to find reduced computing networks . The system can be configured to operate in this manner to reduce the overall latency of the final basic neural network architecture.

In addition, the candidate architecture search space from which the system selects the underlying neural network architecture can be expanded to expand more likely to be accurate and with reduced inference latency on the target computing resource (especially where the target computing resource is data center hardware) In the case of accelerators) the range of available candidate neural networks for execution.

Enlarging the search space as described can increase the number of candidate neural network architectures that are more suitable for data center accelerator deployment, which can lead to a candidate in a search space that may not have not been expanded according to aspects of the present invention The basic neural network architecture identified by one of them. In instances where target computing resources specify hardware accelerators such as GPUs and TPUs, the search space may be augmented using candidate architectures or portions of architectures, such as components or operations that promote operational strength, parallelism, and/or execution efficiency.

In one exemplary augmentation, the search space may be augmented to include neural network architecture components with layers implementing one of various different types of excitation functions. In the case of TPUs and GPUs, it has been found that firing functions such as ReLU or swish are typically of low operational intensity, and are instead typically memory-bound on these types of hardware accelerators. Since the execution of excitation functions in a neural network is typically limited by the total amount of memory available on the target computing resource, the execution of such functions can have a large negative performance impact on the end-to-end network inference speed.

One exemplary augmentation of the search space with respect to excitation functions is to introduce discrete convolutional fusions associated with excitation functions and their equivalents into the search space. Since firing functions are typically operated element-wise and run on a hardware accelerator unit configured for vector operations, execution of these firing functions can be performed in parallel with discrete convolutions, which are typically tied to a hardware Matrix-based operations that operate on the accelerator's matrix unit. These fused excitation function-convolution operations may be selected by the system as a candidate neural network component as part of a search for a basic neural network architecture as described herein. Any of a variety of different excitation functions can be used, including Swish, ReLU (Revised Linear Unit), Sigmoid, Tanh, and Softmax.

Different components including the fusion excitation function-convolution operation layer can be added to the search space and can vary depending on the type of excitation function employed. For example, one component of the excitation function-convolution layer may contain the ReLU excitation function, while the other component may contain the Swish excitation function. It has been found that different hardware accelerators can perform more efficiently using different excitation functions, thus expanding the search space to include fused excitation functions-convolutions of multiple types of excitation functions can further improve the identification of the neural network most suitable for implementation of the discussed neural network. One of the tasks is the basic neural network architecture.

In addition to the components described herein with various different excitation functions, the search space can also be augmented with other fused convolutional structures to further enrich the search space with convolutions of different shapes, types, and sizes. Different convolutional structures may be components added as part of a candidate neural network architecture, and may include 1x1 convolutional expansion layers, depth convolutions, 1x1 convolutional projection layers, and other operations such as excitation functions, batch normalization functions, and/or skip connection.

Identifying the root cause of the non-proportional relationship between computational requirements and latency, as described herein, also demonstrates the impact of parallelism on hardware accelerators. Parallelism can be critical for neural networks implemented on hardware accelerators such as GPUs and TPUs because such hardware accelerators can require large parallelisms to achieve high performance. For example, a convolution operation of a neural network layer needs to have depth, batch, and spatial dimensions of sufficient size to provide sufficient parallelism to achieve high execution efficiency E on matrix units of hardware accelerators. The execution efficiency as described in reference (3) forms part of the overall picture of factors affecting network latency in inference.

Therefore, the search space of the NAS can be augmented in other ways to include operations that can take advantage of the parallelism available on a hardware accelerator. The search space may include one or more operations to fuse depth convolutions with adjacent 1x1 convolutions and operations to reshape the input of a neural network. For example, the input to a candidate neural network can be a scalar. A scalar is a data structure that can represent values according to different orders. For example, a first-order tensor can be a vector, a second-order tensor can be a matrix, and a third-order matrix can be a three-dimensional matrix, and so on. Fusing deep convolutions can be beneficial because deep operations typically have a lower operating intensity, and fusing operations with adjacent convolutions can increase the operating intensity closer to the maximum capability of a hard accelerator.

，則一或多個所添加操作可將張量重塑為

之一維度。 The search space can also include operations that reshape an input tensor by changing different dimensions of the tensor. For example, if a tensor has a depth, width, and height dimensions, one or more operations in the search space that may form part of a candidate neural network architecture may be configured to change the depth, width, and resolution of the tensor One or more of the dimensions. In one example, the search space can be augmented using one or more space-to-depth convolutions, such as 2x2 convolutions, by increasing the depth of the input tensors while decreasing the depth of the input tensors other dimensions to reshape the input tensor. In some implementations, one or more operations are included using stride- n nxn convolutions, where n represents a positive integer that the system can use to reshape a scalar input. For example, if a tensor system input to a candidate neural network

, then one or more of the added operations reshape the tensor to

one dimension.

Implementing the space-to-depth convolution as described above can have at least two advantages. First, convolution is associated with relatively high operational intensity and execution efficiency, and thus more convolution options in the search space can enrich the overall search space when searching for a candidate neural network. High operational intensity and execution efficiency also facilitate implementation on hardware accelerators such as TPUs and GPUs. Then, stride nxn convolutions can also be trained as part of the candidate neural network to contribute to the capacity of the neural network.

The search space may also include operations that reshape the input tensor by moving elements of the tensor to different locations in memory on the target computing resource. Additionally or alternatively, these operations may copy elements to different locations in memory.

In some implementations, the system is configured to directly receive a base neural network for scaling, and does not perform NAS or any other searches for identifying base neural networks. In some implementations, multiple devices individually perform at least a portion of process 200, eg, by identifying a base neural network on one device and scaling the base neural network as described herein on another device.

The system may identify a plurality of scaling parameter values for scaling the base neural network based on the information specifying the target computing resource and the plurality of scaling parameters, as shown in block 240 of FIG. 2 . The system may use delay-aware compound scaling (as presently described) for using the accuracy and delay of a candidate scaling neural network as a target for searching the scaling parameter values for the base neural network.

In general, scaling techniques are applied in conjunction with NAS to identify a neural network that is scaled for deployment on a target computing resource. Model scaling can be used in conjunction with NAS to more efficiently search for a family of neural networks supporting a variety of different use cases. Under a scaling method, various techniques can be used to search for different values of scaling parameters, such as the depth, width, and resolution of a neural network. Scaling can be performed by searching for the value of each scaling parameter separately or by searching for a consistent set of values for adjusting multiple scaling parameters together. The former is sometimes called simple scaling, and the latter is sometimes called compound scaling.

Scaling techniques that use accuracy as the sole objective can result in neural networks that are not scaled when deployed on dedicated hardware, such as data center accelerators, to properly account for the performance/speed impact of such scaled networks. As described in more detail in Figure 3, LACS can use both accuracy and latency goals, which can be shared as the same goal for identifying the underlying neural network architecture.

FIG. 3 is an exemplary procedure 300 for delay-aware compound scaling for a basic neural network architecture. The exemplary process 300 may be executed on a system or device, one or more processors, in one or more locations. For example, procedure 300 may be performed on a NAS-LACS system as described herein.

之候選元組，各具有三個縮放係數。各元組中之係數可為數值，例如，整數或實值。 As shown in block 310, the system selects a plurality of candidate scaling parameter values. The scaling parameter values are the values of different scaling parameters of a neural network. As described herein, depth, width, and input resolution may be scaling parameters, at least because these parameters of a neural network may be larger or smaller. As part of selecting a scaling parameter value, the system may select a tuple of scaling coefficients from a coefficient search space. The coefficient search space contains tuples of candidate scaling coefficients from which scaling parameter values for a neural network can be determined. The number of scaling factors for each tuple may depend on the number of scaling parameters. For example, if the scaling parameters are depth, width, and resolution, the coefficient search space will contain the form

The candidate tuples each have three scaling factors. The coefficients in each tuple can be numeric, eg, integers or real values.

雖然經執行以識別基本神經架構之NAS之間的目標(本文中在(1)處展示)相同於在(2)中展示之目標，但在(2)下運算效能量度之一關鍵差異係評估候選經縮放神經網路 m _scaled 而非基本神經網路 m。系統之總體目標可為識別多個目標之各者之柏拉圖平衡中之縮放參數值，例如，精度及延時。 In a compound scaling method, the scaling factors for each scaling parameter are searched together. The system may apply any of a variety of different search techniques for identifying a coefficient tuple, such as a Platonic leading edge search or a grid-like search. However, the system searches for a tuple of coefficients that the system may search for according to the same goals used to identify the basic neural network architecture described herein with reference to FIG. 2 . Additionally, multiple goals can include both accuracy and latency, which can be expressed as:

While the goal between the NAS performed to identify the underlying neural architecture (shown herein at (1)) is the same as the goal shown in (2), one key difference in the measure of computational efficiency under (2) is to assess The candidate scaled neural network m _scaled instead of the base neural network m . The overall goal of the system may be to identify scaled parameter values in the Platonic equilibrium of each of the multiple goals, eg, accuracy and latency.

In other words, the system determines an efficacy measure of the underlying neural network scaled according to the plurality of candidate neural scaling parameter values, as shown in block 320 . The system determines a performance metric for the base neural network based on different performance metrics reflecting the performance of the candidate scaled neural network scaled from the candidate neural network scaling parameter values.

The basic neural network architecture is scaled by its scaling parameters according to the tuple of scaling coefficients searched in the coefficient search space. ACCURACY( _mscaled ) may be a measure of the accuracy of the candidate scaled neural network on a set of validation instances obtained from received training data. LATENCY( _mscaled ) may be a measure of the time between receiving an input on a candidate scaled neural network and producing a corresponding output when deployed on a target computing resource. In general, the system strives to maximize the accuracy of a candidate neural network architecture while minimizing latency.

According to the description of reference (3), the system can directly or indirectly obtain the performance metrics of the operation strength, operation requirement and execution efficiency of the candidate scaled neural network. This is because LATENCY is based on these three other potential efficacy measures. Thus, in addition to the accuracy and latency of the scaled neural network architecture when deployed on the target computing resource, the system can also be configured to search for candidates that optimize one or more of operational intensity, computational requirements, and execution efficiency Scale parameter value.

As part of determining the efficacy measure, the system may use the received training data to further train or tune the candidate scaled neural network architecture. According to block 330, the system may determine the effectiveness metric of the trained and scaled neural network, and determine whether the effectiveness metric meets a performance threshold. The efficacy measure and efficacy threshold, respectively, may be a composite of multiple efficacy measures and efficacy thresholds. For example, the system can determine a single performance metric from both the scaled neural network's precision and inference latency metrics, or the system can determine separate performance metrics for different targets and compare each metric to a corresponding performance threshold.

If the efficacy metric meets the efficacy threshold, routine 300 ends. Otherwise, the process continues, and according to block 310, the system selects a new plurality of candidate scaling parameter values. For example, the system may select a new tuple of scaling coefficients from the coefficient search space based at least in part on previously selected candidate tuples and their corresponding effectiveness metrics. In some implementations, the system searches multiple coefficient tuples and performs a finer-grained search near each of the candidate tuples according to multiple objectives.

The system may implement any of a variety of different techniques for iteratively searching the coefficient candidate space, such as using gridded search, reinforcement learning, evolutionary search, and the like. The system may continue to search for scaling parameter values until a stopping criterion, such as convergence or number of iterations, is reached, as previously described.

In some implementations, the system may be tuned according to one or more controller parameter values, which may be manually tuned, learned by a machine learning technique, or a combination of both. Controller parameters may affect the relative impact of each target on the overall efficacy measure of a candidate tuple. In some examples, a particular value or relationship between values in a candidate tuple may be favorable or unfavorable based on the learned properties of the ideal scaling factor reflected at least in part in the controller parameter values.

According to block 340, the system generates one or more groups of scaling parameter values from the selected candidate scaling parameter values according to one or more target tradeoffs. The target trade-offs represent different thresholds for each target, such as accuracy and latency, and can be satisfied by different scaled neural networks. For example, a target trade-off may have a higher network accuracy threshold, but a lower inferred latency threshold (ie, a more accurate network with higher latency). As another example, a target trade-off may have a lower network accuracy threshold, but a higher inferred latency threshold (ie, a less accurate network with lower latency). As another example, a target trade-off may balance accuracy and latency performance.

For each target trade-off, the system can identify a respective group of scaling parameter values that the system can use to scale the base neural network architecture to meet the target trade-off. In other words, the system may repeat the selection as shown in block 310, the determination of the efficacy metric as shown in block 320, and the determination of whether the efficacy metric meets performance thresholds as shown in block 330, where the difference between the performance thresholds is determined by the target Balance to define. In some implementations, according to block 330, instead of searching the base neural network, the system may search for candidate scaling factor tuples of the base neural network architecture scaled according to the selected candidate of scaling parameter values that initially satisfy the efficacy measure of the multiple objectives A tuple of road architectures.

，系統可例如藉由一公因數或「複合係數」來縮放元組以獲得用於藉由其他因數來縮放基本神經網路之元組

及

。在此方法中，可快速產生一模型系列，且其可包含適用於各種使用情況之不同神經網路。 In some implementations, after a plurality of scaling parameter values are selected and the efficacy metric is satisfied, the system can be configured to generate a series of neural networks by rescaling the initially scaled neural network. In other words, from a tuple of scaling coefficients

, the system may, for example, scale the tuples by a common factor or "composite factor" to obtain tuples for scaling the base neural network by other factors

and

. In this approach, a family of models can be generated quickly and can include different neural networks suitable for various use cases.

及一縮放係數元組

，一系列中之一經縮放神經網路之縮放參數可藉由以下定義：

其中

分別為神經網路之深度、寬度及解析度之縮放參數值。在第一經縮放神經網路架構之情況中，

可為1。一般言之，複合係數可表示用於網路縮放之延時預算，而

控制如何將延時預算分別分配給不同縮放參數值。 For example, different neural networks in a series can be scaled according to one or more composite coefficients. Given a compound coefficient

and a tuple of scaling coefficients

, the scaling parameters of a scaled neural network in a series can be defined by:

in

are the scaling parameter values of the depth, width and resolution of the neural network, respectively. In the case of the first scaled neural network architecture,

Can be 1. In general, the composite factor can represent the delay budget for network scaling, while

Controls how the delay budget is allocated separately to different scaling parameter values.

Returning to FIG. 2 , the NAS-LACS system may use the architecture of the base neural network scaled according to a plurality of scaling parameter values to generate one or more architectures of the scaled neural network, as shown in block 250 . The scaled neural network architecture may be part of a series of neural networks generated from the base neural network architecture and different scaling parameter values.

If the information specifying the target computing resources includes one or more sets of multiple target computing resources, for example, a plurality of different types of hardware accelerators, the system may repeat the process 200 and the process 300 for each hardware accelerator to generate a corresponding One of the target sets each scales one of the architectures of the neural network. For each hardware accelerator, the system can generate a series of processed algorithms based on different target trade-offs between latency and accuracy or latency, accuracy, and other goals (in particular, including operational intensity and execution efficiency, described herein with reference to (3)). Scaling neural network architectures.

In some implementations, the system can generate multiple series of scaled neural network architectures from the same basic neural network architecture. This approach can help in situations where different target devices share similar hardware characteristics, and can result in faster identification of a corresponding scaled series of devices, at least because the search for a basic neural network architecture is performed only once. Exemplary system:

FIG. 4 is a block diagram of a neural architecture search latency-aware composite scaling (NAS-LACS) system 400 according to an aspect of the present invention. System 400 is configured to receive training data 401 for executing a neural network and target computing resource data 402 specifying target computing resources. System 400 can be configured to implement techniques for generating a series of scaled neural network architectures, as described herein with reference to FIGS. 1-3 .

System 400 can be configured to receive input data according to a user interface. For example, system 400 may receive data as part of a call to an application programming interface (API) of exposure system 400 . System 400 may be implemented on one or more computing devices, as described herein with reference to FIG. 5 . For example, input to system 400 may be through a storage medium (including remote storage connected to one or more computing devices through a network) or as a user on a client computing device coupled to system 400 provided by the input of the interface.

System 400 can be configured to output a scaled neural network architecture 409, such as a series of scaled neural network architectures. The scaled neural network architecture 409 can be sent as output, eg, for display on a user display, and optionally visualized according to the shape and size of each neural network layer as defined in the architecture. In some implementations, the system 400 can be configured to provide the scaled neural network architecture 409 as a set of computer-readable instructions, such as one or more computer programs, which, or the like, can be executed by a target computing resource to implement the Scaling Neural Network Architecture 409.

A computer program can be written in any type of programming language and according to any programming paradigm, such as declarative, procedural, compositional, object-oriented, data-oriented, functional, or imperative. A computer program can be written to perform one or more different functions and operate within a computing environment (eg, on a physical device, a virtual machine, or across multiple devices). A computer program may also implement the functionality described in this specification, such as performed by a system, engine, module or model.

In some implementations, the system 400 is configured to forward data from the scaled neural network architecture 409 to one or more other devices that are configured to convert the architecture to a computer An executable program written in a programming language and optionally as part of a framework for generating machine learning models. System 400 can also be configured to send data corresponding to scaled neural network architecture 409 to a storage device for storage and subsequent retrieval.

System 400 may include a NAS engine 405 . NAS engine 405 and other components of system 400 may be implemented as one or more computer programs, specially configured electronic circuits, or any combination of the foregoing. NAS engine 405 can be configured to receive training data 401 and target computing resource data 402 and generate a base neural network architecture 407 that can be sent to a LACS engine 415 . NAS engine 405 may implement any of a variety of different techniques for neural architecture search described herein with reference to FIGS. 1-3. A system can be configured according to aspects of the present invention to perform NAS using multiple targets, including the inference latency and accuracy of a candidate neural network when executing on the target computing resources. As part of determining that NAS engine 405 can be used to search performance metrics for the underlying neural network architecture, system 400 can include a performance measurement engine 410 .

Performance measurement engine 410 may be configured to receive a framework for a candidate base neural network and generate performance metrics according to goals for performing NAS by NAS engine 405 . The performance metric may provide an overall performance metric for one of the candidate neural networks according to multiple objectives. To determine the accuracy of the candidate base neural network, the performance measurement engine 410 may execute the candidate base neural network on a validation set of one of the training instances, eg, by retaining some training data 401 to obtain the validation set.

To measure latency, performance measurement engine 410 may communicate with a computing resource corresponding to the target computing resource specified by data 402 . For example, if the target computing resource data 402 specifies a TPU as a target resource, the performance measurement engine 410 may send a candidate basic neural network to execute on a corresponding TPU. The TPU may be housed, for example, in a data center in communication with one or more processors implementing system 400 through a network as described in more detail with reference to FIG. 5 .

The performance measurement engine 410 may receive delay information indicative of the delay between receiving the input and generating the output by the target computing resource. The latency information can be measured directly on the target computing resource on-site and sent to the performance measurement engine 410, or measured by the performance measurement engine 410 itself. If the performance measurement engine 410 measures latency, the engine 410 can be configured to compensate for latency not attributable to processing candidate base neural networks, such as network latency for communication to and from the target computing resource. As another example, the performance measurement engine 410 may estimate the latency of processing the input through the candidate base neural network based on previous measurements of the target computing resource and hardware characteristics of the target computing resource.

The performance measurement engine 410 may generate performance metrics of other characteristics of the candidate neural network architecture, such as its operational strength and its execution efficiency. As described herein with reference to FIGS. 1-3 , inference latency can be determined as a function of operational requirements (FLOPS), execution efficiency, and operational intensity together, and in some implementations, system 400 directly or indirectly searches for and Scaling a neural network.

Once the performance metric is generated, the performance measurement engine 410 can send the metric to the NAS engine 405, which can then iteratively search for a new one of the new candidate base neural network architectures until reaching as described herein with reference to FIG. 2 Stop Criteria.

In some examples, tuning the NAS engine 405 according to one or more controller parameters is used to adjust how the NAS engine 405 selects the next candidate base neural network architecture. The controller parameters can be manually tuned according to the desired characteristics of a neural network for a particular neural network task. In some examples, the controller parameters may be learned through any of a variety of machine learning techniques, and the NAS engine 405 may implement one or more machine learning models trained for parameters such as latency and Multiple goals of accuracy to select a basic neural network architecture. For example, NAS engine 405 may implement a recurrent neural network trained to use features of a previous candidate base neural network and targets to predict a candidate base network that is more likely to satisfy the target . A neural network can be trained using performance metrics and training data that are labeled to indicate the final base neural architecture selected in view of a set of training data and target computing resource data associated with a neural network task.

LACS engine 415 may be configured to perform delay-aware compound scaling described in accordance with aspects of this disclosure. LACS engine 415 is configured to receive data 407 from NAS engine 405 specifying a basic neural network architecture. Similar to NAS engine 405, LACS engine 415 may communicate with performance measurement engine 410 to obtain performance metrics for a candidate scaled neural network architecture. The LACS engine 415 can maintain a search space in memory for different scale factor tuples, and can also be configured to scale a final scaled architecture to quickly obtain a range of scaled neural network architectures, as referenced herein 1 to Figure 3 are described. In some implementations, the LACS engine 415 is configured to perform other forms of scaling (eg, simple scaling), but using multiple targets (including latency) used by the NAS engine 405 .

FIG. 5 is a block diagram of an exemplary environment 500 for implementing the NAS-LACS system 400 . System 400 may be implemented on one or more devices having one or more processors in one or more locations, such as in server computing device 515 . Client computing device 512 and server computing device 515 may be communicatively coupled to one or more storage devices 530 through a network 560 . The storage device(s) 530 may be one of a combination of volatile and non-volatile memory, and may be at the same or a different physical location than the computing devices 512, 515. For example, storage device(s) 530 may comprise any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, solid state drive, tape drive, optical storage, memory card, ROM, RAM, DVD , CD-ROM, writable and read-only memory.

Server computing device 515 may include one or more processors 513 and memory 514 . Memory 514 may store information accessible by processor(s) 513 , including instructions 521 executable by processor(s) 513 . Memory 514 may also include data 523 that may be retrieved, manipulated, or stored by processor(s) 513 . Memory 514 may be a type of non-transitory computer-readable medium capable of storing information accessible by processor(s) 513 , such as volatile and non-volatile memory. Processor(s) 513 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FGPAs), and/or application-specific integrated circuits (ASICs), such as tensors Processing Unit (TPU).

Instructions 521 may include one or more instructions that, when executed by processor(s) 513, cause one or more processors to perform the actions defined by the instructions. Instructions 521 may be stored in object code format for direct processing by processor(s) 513, or in other formats, including on-demand or precompiled interpretable instruction code or sets of independent source code modules. Instructions 521 may include instructions for implementing system 400 consistent with aspects of the present invention. System 400 may be executed using processor(s) 513 and/or using other processors located remotely from server computing device 515 .

Data 523 may be retrieved, stored or modified according to instructions 521 by processor(s) 513 . Data 523 may be stored in a computer register, a relational or non-relational database, as a table with a plurality of distinct fields and records or as JSON, YAML, proto or XML documents. Data 523 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 523 may contain information sufficient to identify relevant information, such as numbers, descriptive text, proprietary code, indicators, references to data stored in other memory (including other network locations), or by a function Information used to calculate the relevant data.

The client computing device 512 can also be configured similarly to the server computing device 515 , having one or more processors 516 , memory 517 , instructions 518 and data 519 . The client computing device 512 may also include a user output 526 and a user input 524 . User input 524 may include any suitable mechanism or technology for receiving input from a user, such as keyboards, mice, mechanical actuators, soft actuators, touch screens, microphones, and sensors.

Server computing device 515 can be configured to transmit data to client computing device 512, and client computing device 512 can be configured to display at least a portion of the received data on a display implemented as part of user output 526. part. User output 526 may also be used to display an interface between client computing device 512 and server computing device 515 . Alternatively or in addition, the user output 526 may include one or more speakers, transducers or other audio outputs, a haptic interface or other haptics that provide non-visual and non-auditory information to the platform user of the client computing device 512 Give back.

Although Figure 5 shows processors 513, 516 and memories 514, 517 as being within computing devices 515, 512, the components described in this specification, including processors 513, 516 and memories 514, 517, may include Multiple processors and memories in different physical locations and not operating within the same computing device. For example, some instructions 521, 518 and data 523, 519 may be stored on a removable SD card, and others may be stored on a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from the processors 513 , 516 but still accessible by the processors 513 , 516 . Similarly, processors 513, 516 may comprise sets of processors that may perform simultaneous and/or sequential operations. The computing devices 515, 512 may each include one or more internal clocks that provide timing information that may be used for time measurement of operations and programs run by the computing devices 515, 512.

The server computing device 515 may be connected through a network 560 to a data center 550 that houses the hardware accelerators 551A-N. Data center 550 may be one of multiple data centers or other facilities in which various types of computing devices, such as hardware accelerators, are located. The computing resources housed in the data center 550 may be designated as part of the target computing resources for deploying the scaled neural network architecture, as described herein.

Server computing device 515 may be configured to receive requests from client computing device 512 to process data on computing resources in data center 550 . For example, environment 500 may be part of a computing platform that is configured to provide various services to users through various user interfaces and/or APIs that expose platform services. One or more services may be a machine learning framework or set of tools for generating neural networks or other machine learning models from a given task and training data. Client computing device 512 may receive and transmit data specifying target computing resources to be allocated for executing a neural network that is trained to perform a particular neural network task. According to aspects of the invention described herein with reference to FIGS. 1-4, NAS-LACS system 400 may receive data and training data specifying a target computing resource, and in response generate a series of processed data for deployment on the target computing resource. Scaling neural network architectures.

As other examples of potential services provided by a platform of implementation environment 500, server computing device 515 may maintain various sets of scaled neural network architectures according to different potential target computing resources available at data center 550. For example, server computing device 515 may maintain different families for deploying neural networks on various types of TPUs and/or GPUs housed in data center 550 or otherwise available for processing.

Devices 512 , 515 and data center 550 may be capable of direct and indirect communication over network 560 . For example, using a network socket, client computing device 512 may connect to a service operating in data center 550 via an Internet protocol. Devices 515, 512 can set up listening sockets that can accept an initial connection for sending and receiving information. The network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, Intranets, Virtual Private Networks, Wide Area Networks, Local Area Networks, and private networks using one or more company-proprietary communication protocols. network. Network 560 may support various short-range and long-range connections. Short-range and long-range connections can be made on different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol); or use various communications standards, such as the LTE® standard for wireless broadband communications. Additionally or alternatively, network 560 may also support wired connections between devices 512, 515 and data center 550, including through various types of Ethernet connections.

Although a single server computing device 515, client computing device 512, and data center 550 are shown in FIG. 5, it should be understood that aspects of the present invention may be implemented with various configurations and numbers of computing devices, including using In the case of sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of the present invention may be performed on a single device connected to a hardware accelerator configured to process neural networks, and any combination thereof, among others. Exemplary use case:

As described herein, aspects of the present invention provide an architecture for scaling a neural network from a base neural network according to a multi-objective approach. The following are examples of neural network tasks.

As an example, the input to the neural network may be in the form of images, video. As part of processing a given input, such as as part of a computer vision task, a neural network can be configured to extract, identify and generate features. A neural network trained to perform this type of neural network task can be trained to generate an output classification from a set of different potential classifications. Additionally or alternatively, the neural network may be trained to output a score corresponding to an estimated probability that an identified object in an image or video belongs to a particular class.

As another example, the input to the neural network may be data files corresponding to a particular format, such as HTML files, word processing documents, or formatted metadata obtained from other types of data, such as image file metadata. In this context, a neural network task may be to classify, score, or otherwise predict some characteristic about the received input. For example, a neural network can be trained to predict the probability that the received input contains words associated with a particular object. Also as part of performing a particular task, neural networks can be trained to generate text predictions, such as as part of a tool for automatically completing text in a document as it is being written. A neural network can also be trained to predict a translation of text in an input document to a target language, such as when composing a message.

Other types of input files may be data related to the characteristics of a network of interconnected devices. These input files may contain activity logs, as well as records of access privileges for different computing devices to access different sources of potentially sensitive data. A neural network can be trained to process these and other types of documents to predict ongoing and future network security breaches. For example, a neural network can be trained to predict the intrusion of a malicious actor into a network.

As another example, the input to a neural network may be audio input, including streamed audio, pre-recorded audio, and audio that is part of a video or other source or media. One of the neural network tasks in the audio content context may include speech recognition, including isolating speech from other recognized audio sources and/or enhancing characteristics of the recognized speech to be more easily heard. A neural network can be trained to predict an accurate translation of input speech to a target language, eg, in real-time as part of a translation tool.

In addition to data inputs comprising various types of data described herein, a neural network can also be trained to process features corresponding to a given input. A feature is a value (eg, a value or category) associated with some characteristic of the input. For example, in the context of an image, a feature of the image may be related to the RGB value of each pixel in the image. A neural network task in the image/video content context may be to classify the content of an image or video, eg, for the presence of different people, places, or things. Neural networks can be trained to extract and select relevant features for processing to generate an output for a given input, and can also be trained to generate new features based on learned relationships between various properties of the input data.

Aspects of the invention can be implemented in digital circuitry, in computer-readable storage media, as one or more computer programs, or in a combination of one or more of the foregoing. A computer-readable storage medium may be non-transitory, eg, as one or more instructions executable by the processor(s) and stored on a tangible storage device.

In this specification, the phrase "configured to" is used in various contexts relating to computer systems, hardware, or parts of a computer program. When a system is said to be configured to perform one or more operations, it means that the system has the appropriate software, firmware and/or hardware installed on the system, which software, firmware and/or hardware is Causes the system to perform one or more operations when in operation. When a piece of hardware is said to be configured to perform one or more operations, it means that the piece of hardware includes one or more circuits that, when in operation, receive inputs and, in accordance with the inputs, correspond to One or more operations produce output. When a computer program is said to be configured to perform one or more operations, it means that the computer program includes one or more program instructions that, when executed by one or more computers, result in an or more computers to perform one or more operations.

Although the operations shown in the drawings and described in the scope of the invention are shown in a particular order, it should be understood that the operations may be performed in a different order than shown, and that some operations may be omitted, performed more than once, and / or in parallel with other operations. Furthermore, separation of different system components configured to perform different operations should not be construed as requiring separation of components. The described components, modules, programs and engines may be integrated together as a single system, or may be part of multiple systems.

Unless otherwise indicated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above-discussed features may be utilized without departing from the subject matter defined by the scope of the invention, the preceding description of the embodiments should be considered by way of illustration and not by way of Limitations are made by the subject matter defined by the scope of the invention application. In addition, the provision of examples described herein and clauses expressed as "such as", "including" and the like should not be construed as limiting the subject matter of the patentable scope of the invention to the particular examples; one of many possible forms. Furthermore, the same reference numbers in different figures may identify the same or similar elements.

101: Basic Neural Network Architecture 103: Series 104A to N: Scaled Neural Network Architecture 107A to N: Candidate Nets 108: Coefficient Search Space 109: Scaled Neural Network Architecture 115:Data Center 116: Hardware Accelerator 200: Program 210: Blocks 220: Square 230: Square 240: Square 250: Square 300: Procedure 310: Blocks 320: Square 330: Square 340: Square 400: Neural Architecture Search Latency-Aware Composite Scaling (NAS-LACS) System 401: Training Materials 402: Target computing resource information 405: Neural Architecture Search (NAS) Engine 407: Basic Neural Network Architecture/Material 409: Scaled Neural Network Architecture 410: Efficiency Measurement Engine 415: Latency-Aware Composite Scaling (LACS) Engine 500: Environment 512: Client computing device 513: Processor 514: Memory 515: Server computing device 516: Processor 517: Memory 518: Command 519: Information 521: Command 523: Information 524: User input 526: user output 530: Storage Device 550:Data Center 551A to N: Hardware Accelerator 560: Internet

1 is a block diagram illustrating a series of scaled neural network architectures for deployment in a data center that houses a hardware accelerator upon which the deployed neural network will execute.

2 is a flow diagram of an exemplary process for generating a scaled neural network architecture for execution on a target computing resource.

3 is an exemplary procedure for delay-aware compound scaling for a basic neural network architecture.

4 is a block diagram of a neural architecture search latency-aware complex scaling (NAS-LACS) system according to an aspect of the present invention.

5 is a block diagram of an exemplary environment for implementing a NAS-LACS system.

101: Basic Neural Network Architecture

103: Series

104A to N: Scaled Neural Network Architecture

107A to N: Candidate Nets

108: Coefficient Search Space

109: Scaled Neural Network Architecture

115:Data Center

116: Hardware Accelerator

Claims (20)

A computer-implemented method for determining an architecture of a neural network, comprising: receive information specifying target computing resources by one or more processors; receiving, by the one or more processors, data specifying an architecture of a basic neural network; identifying, by the one or more processors, a plurality of scaling parameter values for scaling the basic neural network according to the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network, wherein the identifying Include repeating the following steps: selecting a plurality of candidate scaling parameter values, and determining an efficacy metric of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein the efficacy metric is determined according to a plurality of targets including a delay target; and An architecture of a scaled neural network is generated by the one or more processors using the architecture of the base neural network scaled according to the plurality of scaling parameter values. As in the method of claim 1, wherein the plurality of targets are a plurality of second targets; and wherein receiving the data specifying the architecture of the underlying neural network includes: receiving, by one or more processors, training data corresponding to a neural network task, and The architecture of the base neural network is identified by the one or more processors using the training data and performing a neural architecture search on a search space according to a plurality of first targets. As in the method of claim 2, wherein the search space includes candidate neural network layers, each candidate neural network layer configured to perform one or more respective operations, and Wherein the search space includes candidate neural network layers comprising different respective excitation functions. As in the method of claim 3, wherein the architecture of the basic neural network includes a plurality of components, each component has a respective plurality of neural network layers, and Wherein the search space includes a plurality of candidate components of the candidate neural network layer, including a first component of the candidate network layer, which includes a first excitation function; and a second component of the candidate network layer, which includes a different One of the first excitation functions is a second excitation function. The method of claim 2, wherein the plurality of first targets used to perform the neural architecture search are the same as the plurality of second targets used to identify the plurality of scaling parameter values. The method of claim 2, wherein the plurality of first targets and the plurality of second targets include an accuracy target corresponding to the accuracy of the output of the base neural network when training using the training data. The method of claim 1, wherein when the base neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources, the performance metric is at least partially associated with the base neural network to receive an input and A delay metric correspondence between an output is generated. The method of claim 1, wherein when the basic neural network is deployed on the target computing resources, the delay target corresponds to a minimum delay between the basic neural network receiving an input and generating an output. As in the method of claim 1, wherein the information specifying the target computing resources specifies one or more hardware accelerators; and Wherein the method further includes executing the scaled neural network on the one or more hardware accelerators to perform the neural network task. According to the method of claim 9, wherein the target computing resources are first target computing resources, the plurality of scaling parameter values are a plurality of first scaling parameter values, and Wherein the method further includes: receiving, by the one or more processors, information specifying a second target computing resource different from the first target computing resources, and A plurality of second scaling parameter values for scaling the base neural network are identified based on the information specifying the second target computing resources, wherein the plurality of second scaling parameter values are different from the plurality of first scaling parameter values . The method of claim 1, wherein the plurality of scaling parameter values are a plurality of first scaling parameter values, and wherein the method further includes generating a scaled neural network architecture from the base neural network architecture scaled using a plurality of second scaling parameter values, wherein the first scaling parameters are modified in accordance with the plurality of first scaling parameter values and consistently One or more composite coefficients of each of the values yield the second scaling parameter values. The method of claim 1, wherein the base neural network is a convolutional neural network, and wherein the plurality of scaling parameters include a depth of the base neural network, a width of the base neural network, and a base neural network One or more of the input resolutions. A system comprising: one or more processors, and One or more storage devices, or the like, are coupled to the one or more processors and store instructions that, when executed by the one or more processors, cause the one or more processors to execute for determining The operation of an architecture of a neural network, including: Receive information about the specified target computing resource; receive data specifying an architecture of a basic neural network; Identifying a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network, wherein the identifying includes repeating the following steps: selecting a plurality of candidate scaling parameter values, and determining an efficacy metric of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein the efficacy metric is determined according to a plurality of targets including a delay target; and An architecture of a scaled neural network is generated using the architecture of the base neural network scaled according to the plurality of scaling parameter values. The system of claim 13, wherein the plurality of targets are a plurality of second targets; and wherein receiving the data specifying the architecture of the underlying neural network includes: receiving training data corresponding to a neural network task, and A neural architecture search is performed on a search space using the training data and according to a plurality of first targets to identify the architecture of the base neural network. As in the system of claim 14, wherein the search space includes candidate neural network layers, each candidate neural network layer configured to perform one or more respective operations, and Wherein the search space includes candidate neural network layers comprising different respective excitation functions. The system of claim 14, wherein the plurality of first targets used to perform the neural architecture search are the same as the plurality of second targets used to identify the plurality of scaling parameter values. The system of claim 14, wherein the plurality of first targets and the plurality of second targets include an accuracy target corresponding to the accuracy of the output of the base neural network when trained using the training data. The system of claim 13, wherein when the base neural network is scaled according to the plurality of candidate scaling parameter values and deployed on the target computing resources, the performance measure is at least partially associated with the base neural network receiving an input and A delay metric correspondence between an output is generated. The system of claim 13, wherein the delay target corresponds to a minimum delay between the basic neural network receiving an input and generating an output when the basic neural network is deployed on the target computing resources. One or more non-transitory computer-readable storage media, or the like, store instructions that, when executed by one or more processors, cause the one or more processors to execute one of the functions for determining a neural network The operation of the architecture, including: Receive information about the specified target computing resource; receiving, by the one or more processors, data specifying an architecture of a basic neural network; Identifying a plurality of scaling parameter values for scaling the basic neural network based on the information specifying the target computing resources and a plurality of scaling parameters of the basic neural network, wherein the identifying includes repeating the following steps: selecting a plurality of candidate scaling parameter values, and determining an efficacy metric of the base neural network scaled according to the plurality of candidate scaling parameter values, wherein the efficacy metric is determined according to a plurality of targets including a delay target; and An architecture of a scaled neural network is generated using the architecture of the base neural network scaled according to the plurality of scaling parameter values.

Images