JP7578779B2 - Method and system for adapting neural networks to dynamic environments - Google Patents

Description

The present disclosure relates generally to the fields of artificial intelligence and data analytics. More specifically, the present disclosure relates to methods and systems for adapting neural networks to dynamic environments.

A neural network is a computational model that attempts to mimic the way the human brain makes decisions. Neural networks are widely used in a variety of applications across different industries. When used in such applications, neural networks adapt to different types of environments. For example, a neural network implemented in a video analytics system, such as a surveillance application for workplace safety, needs to identify workers wearing safety gear on an assembly line. Also, the neural network needs to identify workers wearing safety gear on a production line. Because the environments of an assembly line and a production line are different, there is a large amount of background variability that affects the identification of workers wearing safety gear. This can cause the neural network to misidentify or miss identifying workers, which reduces the performance of the neural network. In another example, in an object recognition application, a neural network trained using images of objects taken in a sunny area may not work well in a cloudy area. The neural network may not work well under different weather conditions because it may misidentify objects. Therefore, neural networks need to be developed for different environments and their performance improved. However, such development is expensive and tedious. Additionally, the lack of data and annotations in various applications makes it difficult to adapt neural networks to different environments.

Conventional techniques such as semi-supervised learning, zero-shot learning, and few-shot learning address the need for large amounts of data and annotations. However, these techniques do not provide a solution to extend neural networks to deal with different environments without minimal annotation. Moreover, these techniques, despite producing annotations, have low accuracy. Some other conventional techniques utilize image view extraction to improve the performance of neural networks. However, these conventional techniques are unable to adapt to different backgrounds and environments because they only consider foreground features in the data that are relevant to the application of the neural network.

The information disclosed in the Background section of this disclosure is intended to enhance understanding of the general background of the present invention and should not be construed as an admission or in any manner suggesting that this information forms prior art already known to those of skill in the art.

In an embodiment, the present disclosure discloses a method for adapting a neural network to a dynamic environment. The method comprises receiving a first set of features of the input data items associated with a first portion of the input data items. The method further comprises determining a second set of features of the input data items associated with a second portion of the input data items by associating the first set of features with one or more correlation pairs generated using the neural network. The one or more correlation pairs are generated by receiving a feature set including a first set of training features and a second set of training features for each data set of the plurality of training data items from the one or more neural networks. The one or more first labels and the one or more second labels for each data set of the plurality of training data items are generated by projecting the first set of training features and the second set of training features onto a feature space. The first set of training features and the second set of training features are relocated on the feature space based on the correlation between the first set of training features and the second set of training features. One or more correlation pairs, each indicating one or more first labels and corresponding second labels for each of the data sets of the plurality of training data items, are generated based on the rearrangement of the first set of training features and the second set of training features.

In one embodiment, the present disclosure discloses a system for adapting a neural network to a dynamic environment. The system includes one or more processors and a memory. The one or more processors are configured to receive a first set of features of the input data items associated with a first portion of the input data items. Furthermore, the one or more processors are configured to determine a second set of features of the input data items associated with a second portion of the input data items by associating the first set of features with one or more correlation pairs generated using the neural network. The one or more correlation pairs are generated by receiving a feature set including a first set of training features and a second set of training features for each dataset of the plurality of training data items from the one or more neural networks. The one or more first labels and the one or more second labels for each dataset of the plurality of training data items are generated by projecting the first set of training features and the second set of training features onto a feature space. The first set of training features and the second set of training features are relocated on the feature space based on the correlation between the first set of training features and the second set of training features. One or more correlation pairs, each indicating one or more first labels and corresponding second labels for each of the data sets of the plurality of training data items, are generated based on the rearrangement of the first set of training features and the second set of training features.

The foregoing summary is illustrative and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

The novel features and characteristics of the present disclosure are set forth in the appended claims. However, the disclosure itself, as well as its preferred modes of use, further objects and advantages, will be best understood by reference to the following detailed description of illustrative embodiments, when read in conjunction with the accompanying drawings, in which: One or more embodiments are now described, by way of example only, with reference to the accompanying drawings, in which like reference numerals represent like elements throughout.

FIG. 1 illustrates an example environment for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. FIG. 2 is a detailed diagram of a system for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. FIG. 3A is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. FIG. 3B is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. FIG. 3C is an exemplary diagram for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. FIG. 4 is an exemplary flowchart illustrating method steps for training a neural network to adapt to a dynamic environment, according to some embodiments of the present disclosure. FIG. 5 is an exemplary flow chart illustrating method steps for adapting a neural network to a dynamic environment according to some embodiments of the present disclosure. FIG. 6 is a block diagram of a general-purpose computing system for adapting neural networks to dynamic environments according to an embodiment of the present disclosure.

Those skilled in the art will appreciate that any block diagrams herein represent conceptual diagrams of example systems embodying the principles of the present subject matter. Similarly, any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like will be understood to represent various processes that may be substantially represented on a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

While the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are described in detail below. It is to be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

"Include", "comprising" or other variations thereof are intended to be non-exclusive inclusive, and a setup, apparatus or method consisting of a list of components or steps does not include only those components or steps, but may include other components or steps not expressly listed or inherent to such setup or apparatus or method. In other words, a statement "including..." of one or more elements in a system or apparatus does not preclude the presence of other or additional elements in the system or apparatus, absent further constraints.

Neural networks are used in various applications in various industries. When used in such applications, neural networks respond to different types of environments. To obtain better performance, neural networks need to adapt to different types of environments. The present disclosure provides a method and system for adapting a neural network to different dynamic environments. The neural network is trained by providing a first set of training features (e.g., background features) and a second set of training features (e.g., foreground features) of a plurality of training data items. The neural network correlates the first set of features and the second set of features to form a correlation pair with a first label (e.g., background label) and a second label (e.g., foreground label). When the neural network receives the first set of features (e.g., background features) of the input data items, the neural network determines the second set of features (e.g., foreground features) of the input data items based on the correlation pair. Thus, the neural network can adapt to new environments. Thus, the present disclosure enables the neural network to adapt to different environments. The present disclosure can be used to develop neural networks under data scarcity constraints for new environments (e.g., new Internet of Things (IoT) environments), thereby expanding the applications of neural networks.

FIG. 1 illustrates an exemplary environment 100 for adapting a neural network to a dynamic environment, according to some embodiments of the present disclosure. The environment 100 includes a neural network 102 and a system 104. An input data item 101 may be provided to the neural network 102. The input data item 101 includes one of an image, a video, an audio input, a text input, a voice input, and the like. The input data item 101 may be received from a source such as a capture unit, a voice recorder, a database, and the like. The system 104 receives a first set 103 of features of the input data item 101 (first set of features 103) associated with a first portion of the input data item 101. For example, if the input data item 101 is an image, the first portion of the input data item 101 may be a background of the image. In one embodiment, the first set of features 103 may be received from the neural network 102. The neural network 102, also known as an artificial neural network, is a computing system having interconnected nodes. The neural network 102 recognizes hidden patterns and correlations in data, domains, and classifications over time for continuous learning and improvement. The neural network 102 is composed of at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), a recurrent neural network, a graph convolutional network, a sequential neural network, an encoder-decoder network, and combinations thereof. Those skilled in the art will understand that the present disclosure is applicable to any neural network other than the above neural networks.

The system 104 may determine a second set 108 of features of the input data item 101 (second set of features 108) associated with the second portion of the input data item 101 by associating the first set 103 of features with one or more correlation pairs. For example, if the input data item 101 is an image, the second portion of the input data item 101 may be the foreground of the image. As used herein, the foreground of the input data item 101 refers to a portion of the input data item 101 that includes objects of interest/audio features (if the input data item 101 includes an audio input) that are relevant to a given application of the neural network 102. The background of the input data item 101 refers to a portion of the input data item 101/audio features that do not include objects of interest/are not relevant to a given application of the neural network 102. The correlation pairs are generated by receiving a feature set consisting of a first set of training features (first set of training features) and a second set of training features (second set of training features) for each data set of a plurality of training data items using the neural network 102. For example, the first set of training features may be background training features and the second set of training features may be foreground training features. Each dataset of the plurality of training data items may be composed of an original data item and one or more transformations of the original data item. The first set of training features and the second set of training features are projected onto a feature space to generate a first label and a second label. Furthermore, the first set of training features and the second set of training features are rearranged onto the feature space by correlating the first set of training features with the second set of training features. Based on the rearrangement, one or more correlation pairs are generated, each indicating a first label and a corresponding second label. The correlation pairs indicate a first label (e.g., a background label) and a corresponding second label (e.g., a foreground label). When a new input data item 101 having new features (e.g., a background feature) is received, the neural network 102 can determine a foreground feature of the input data item 101 and adapt to a different environment. Neural networks 102 can be implemented in applications such as object detection, object tracking, and face recognition.

The system 104 may include an input/output (I/O) interface 106, a memory 107, and a central processing unit 105 (also referred to as a "CPU" or "one or more processors 105"). In some embodiments, the memory 107 may be communicatively coupled to the one or more processors 105. The memory 107 stores instructions executable by the one or more processors 105. The one or more processors 105 may include at least one data processor for executing program components for executing user or system generated requests. The memory 107 may be communicatively coupled to the one or more processors 105. The memory 107 stores instructions executable by the one or more processors 105 that, when executed, may cause the one or more processors 105 to adapt the neural network 102 to a dynamic environment. The I/O interface 106 is coupled to the one or more processors 105 to which input signals or/and output signals are communicated. For example, an input data item 101 may be received from a user via the I/O interface 106. In one embodiment, the system 104 may be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a personal computer (PC), a notebook, a smartphone, a tablet, a server, a network server, a cloud-based server, etc.

A detailed diagram 200 of a system 104 for adapting a neural network 102 to different environments is shown, according to some embodiments of the present disclosure. In one embodiment, the memory 107 may include one or more modules 202 and data 201. The one or more modules 202 may be configured to perform steps of the present disclosure using the data 201 to adapt the neural network 102 to a dynamic environment. In one embodiment, each of the one or more modules 202 may be a hardware unit that is external to the memory 107 and may be coupled with the system 104. As used herein, the term module 202 refers to an application specific integrated circuit (ASIC), an electronic circuit, a field-programmable gate array (FPGA), a programmable system-on-chip (PSoC), a combinational logic circuit, and/or other suitable components that provide the described functionality. The one or more modules 202, when configured with the described functionality defined in this disclosure, result in novel hardware.

In one embodiment, modules 202 may include, for example, input module 209, label generation module 210, rearrangement module 211, correlation module 212, decision module 213, and other module 214. It will be appreciated that such aforementioned modules 202 may be represented as a single module or as a combination of different modules. In one implementation, data 201 may include, for example, input data 203, label data 204, rearrangement data 205, correlation data 206, decision data 207, and other data 208.

In one embodiment, the input module 209 may be configured to receive a feature set for a dataset of each of the plurality of training data items. The plurality of training data items may include one of an image, a video, an audio input, a text input, a voice input, and the like. Each dataset of the plurality of training data items is comprised of an original data item and one or more transformations of the original data item. The one or more transformations of the original data item may comprise at least one of a rotation, a translation, a scaling, and the like. For example, the dataset may be comprised of an original image, a rotated version of the original image, a scaled version of the original image, and the like. The feature set may be comprised of a first set of training features and a second set of training features for each dataset of the plurality of training data items. The first set of training features may be associated with a first portion of each of the plurality of training data items. The second set of training features may be associated with a second portion of each of the plurality of training data items. For example, consider the training data items to be images. In that case, the first portion of the training data items may be the background of the image. Meanwhile, the second portion of the training data items may be the foreground of the image. The image consists of a train in the foreground and people in a train station in the background. In another example, the image consists of workers in the foreground and an assembly line environment in the background.

In one embodiment, during training of the neural network 102, the input module 209 may receive a first set of training features and a second set of training features from one or more neural networks 102. For example, the first neural network may be used to extract a first set of training features associated with a first portion of a data set of each of the multiple training data items. The second neural network may be used to extract a second set of training features associated with a second portion of a data set of each of the multiple training data items. The one or more neural networks 102 may be trained neural networks or untrained neural networks. In one embodiment, the multiple training data items may be provided to the input module 209. The input module 209 may generate the first and second portions of each of the multiple training data items. For example, the input module 209 may generate the first and second portions of each of the multiple training data items using a background subtraction technique to obtain the foreground and background of each of the multiple training data items. Those skilled in the art will appreciate that any technique other than the techniques described above may be used to obtain the first and second portions of each of the plurality of training data items. In another embodiment, the input module 209 may receive the first and second portions of each of the plurality of training data items. The first and second portions may be received from a user, a database, etc.

Referring to the example 300 of FIG. 3A, a first portion 302 and a second portion 301 of each of the plurality of training data items are illustrated. In this example, each of the plurality of training data items is an image. The first portion 302 is the background of the image, and the second portion 301 is the foreground of the image. Furthermore, one or more transformations 303 of each of the plurality of training data items are generated. Next, one or more neural networks 102 (neural network 102 ₁ , neural network 102 ₂ ) can extract a first set of features and a second set of features of the dataset of each of the plurality of training data items. For example, neural network 102 ₁ generates a second set of features 304 (e.g., foreground features), and neural network 102 ₂ generates a first set of features 305 (e.g., background features). In one example, the neural network 102 may be trained for multiple classes of each of the plurality of training data items based on tasks performed by the neural network 102 in different applications. A first set of features 305 and a second set of features 304 corresponding to each class may be extracted. The feature sets may be stored in memory 107 as input data 203.

In one embodiment, the label generation module 210 may be configured to receive the input data 203 from the input module 209. Furthermore, the label generation module 210 may be configured to generate one or more first labels and one or more second labels for each data set of the plurality of training data items. The label generation module 210 generates one or more first labels (e.g., background labels) and one or more second labels (e.g., foreground labels) by projecting the first set of training features and the second set of training features onto a feature space. First, the label generation module 210 projects the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the training data items from the plurality of training data items. For example, the training data items may be images. The one or more classes of the second portion of the image (e.g., foreground) may include object identification, object recognition, etc. In one example, the neural network 102 may be implemented in pedestrian classification. In that case, the one or more classes may include pedestrian identification, pedestrian recognition, etc. In another example, the neural network 102 may be implemented in a monitoring application for workplace safety. The one or more classes may include worker identification, safety equipment compliance detection, etc. In one embodiment, the feature space may be a hypersphere. Those skilled in the art will appreciate that the first and second sets of training features may be projected into any feature space other than the feature spaces mentioned above. Referring to 306 in FIG. 3B, triangles indicate training features from the second set of training features projected into feature space 308. Similarly, stars indicate training features from the first set of training features projected into feature space 308. The label generation module 210 then creates multiple regions based on the similarity between the first set of training features and the second set of training features. Referring to the example of FIG. 3B, multiple regions 307 ₁₁ , 307 ₁₂ , 307 ₂₁ , 307 ₂₂ , 307 ₃₁ and 307 ₃₂ are created. If the feature space is a hypersphere, the multiple regions are spheres. In other examples, the regions may include a square, an eclipse, etc. 309 denotes a curved hyperplane separating the first set of training features and the second set of training features.

The label generation module 210 may identify a center and a radius for each of the multiple regions. The label generation module 210 then adjusts the radius for each of the multiple regions by projecting the first set of training features and the second set of training features in the feature space for each of the one or more classes of the second portion of the other training data items from the multiple training data items. If the feature space is a hypersphere, the label generation module 210 adjusts the radius based on the following equation (1):

ここで、「W_pxk」は中心行列、「R_pxk」は半径行列、「l_mxn」は訓練特徴の第１及び第２のセット、「Dist」は特徴空間における点（訓練特徴の第１のセット及び訓練特徴の第２のセット）間の距離である。距離は、ユークリッド距離、コサイン距離などを用いて計算することができる。ラベル生成モジュール２１０は、対応する領域に関連付けられた、訓練特徴の第１のセットと訓練特徴の第２のセットの類似性に基づいて、複数の領域の各々に対してラベルを生成する。例えば、「黄色いシャツを着た歩行者」というラベルが、領域３０７_１１に対して生成されることがある。複数の領域の数は、複数の訓練データ項目の各々の１つ以上の変換３０３の数と、１つ以上のクラスの数に基づく。一実施形態では、複数の領域の数は、１つ以上のクラスと１つ以上の変換３０３との積である。１つ以上の第１のラベル及び１つ以上の第２のラベルは、ラベルデータ２０４としてメモリ１０７に記憶されてもよい。

where " _Wpxk " is a centroid matrix, " _Rpxk " is a radius matrix, " _lmxn " is the first and second set of training features, and "Dist" is the distance between points (first set of training features and second set of training features) in feature space. The distance can be calculated using Euclidean distance, cosine distance, etc. The label generation module 210 generates a label for each of the plurality of regions based on the similarity of the first set of training features and the second set of training features associated with the corresponding region. For example, a label "pedestrian wearing yellow shirt" may be generated for region _307-11 . The number of the plurality of regions is based on the number of one or more transformations 303 of each of the plurality of training data items and the number of one or more classes. In one embodiment, the number of the plurality of regions is a product of the one or more classes and the one or more transformations 303. The one or more first labels and the one or more second labels may be stored in the memory 107 as label data 204.

In one embodiment, the rearrangement module 211 may be configured to receive the input data 203 from the input module 209 and the label data 204 from the label generation module 210. Furthermore, the rearrangement module 211 may be configured to rearrange the first set of training features and the second set of training features on the feature space based on the correlation between the first set of training features and the second set of training features. In particular, the rearrangement module 211 correlates the one or more second labels with the one or more first labels by providing the one or more first labels from the one or more neural networks 102 associated with the generation of the second set of training features to the first neural network. Furthermore, the one or more second labels are provided to the second neural network from the one or more neural networks 102 associated with the generation of the first set of training features. The correlation may be performed using a swapped labeling technique. Those skilled in the art will understand that any technique other than the above-mentioned techniques may be used to correlate the one or more second labels with the one or more first labels. The relocation module 211 may relocate the first set of training features and the second set of training features in the feature space based on the correlation. In one example, the relocation may include moving a region having the first set of training features relative to a region having the second set of training features. With reference to the example of FIG. 3B, region _307-12 may be moved closer to region _307-11 based on the correlation. Data related to the relocated first set of training features and the second set of training features may be stored in the memory 107 as relocation data 205.

In one embodiment, the correlation module 212 may be configured to receive the rearrangement data 205 from the rearrangement module 211. Furthermore, the correlation module 212 may be configured to generate one or more correlation pairs. The correlation module 212 may generate one or more correlation pairs based on the rearrangement of the first set of training features and the second set of training features. Each of the one or more correlation pairs may indicate one or more first labels and corresponding second labels of each data set of the plurality of training data items. With reference to the example of FIG. 3B, the first correlation pair includes region 307 ₁₁ and region 307 ₁₂ , the second correlation pair includes region 307 ₂₁ and region 307 ₂₂ , and the third correlation pair includes region 307 ₃₁ and region 307 _32. In one example, the correlation pair may indicate a background label and a corresponding foreground label of each data set of the plurality of images. The one or more correlation pairs may be stored in the memory 107 as correlation data 206.

In one embodiment, during deployment, the input module 209 is further configured to receive a first set 103 of features of the input data item 101. The input data item 101 may include one of an image, a video, a voice input, a text input, and a voice input, and the like. The first set 103 of features may be associated with a first portion of the input data item 101. For example, the input data item 101 may be an image. The first portion of the input data item 101 may be a background of the image. In one example, the background of the image may include a train in an open area. In another example, the input data item 101 may be written text. The background may include blue paper. In another example, the background includes a production line environment. The neural network may be trained in an assembly line environment. In one embodiment, the input module 209 may receive the first set 103 of features from the neural network 102. In another embodiment, the input module 209 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like. The first set of features 103 may be stored in memory 107 as input data 203.

In one embodiment, the determination module 213 is configured to receive the input data 203 from the input module 209. Furthermore, the determination module 213 is configured to determine a second set of features 108 of the input data item 101. The second set of features 108 is associated with a second portion of the input data item 101, e.g., the foreground of the image. The determination module 213 may determine the second set of features 108 by associating the first set of features 103 with one or more correlation pairs. The determination module 213 may associate the first set of features 103 with one or more correlation pairs by generating a transformation matrix Wt 310, as shown in FIG. 3C. In one embodiment, the transformation matrix Wt may be generated using a Riemannian conditioning technique. The determination module 213 may associate the first set of features 103 with one or more correlation pairs using equation (2), as given below:

ここで、Arccosはコサイン関数の逆関数を示す。当業者であれば、上述の関数以外の任意の逆三角関数が、特徴の第１のセット１０３と１つ又は複数の相関ペアとの関連付けに使用され得ることを理解するであろう。特徴の第２のセット１０８は、メモリ１０７に決定データ２０７として記憶されてもよい。上述の例を参照すると、ニューラルネットワーク１０２は、背景に人がいる場合に列車又は列車の周囲の物体を識別するように訓練されてもよい。本開示は、列車がオープンエリアにある場合に、ニューラルネットワーク１０２が列車又は列車の周囲の物体を識別することを可能にする。したがって、本開示は、異なる環境におけるニューラルネットワーク１０２による予測における背景変動（例えば、オープンエリアにおける太陽光）を克服し、したがって、ニューラルネットワーク１０２の性能及びスケーラビリティを向上させる。

Here, Arccos denotes the inverse of the cosine function. Those skilled in the art will understand that any inverse trigonometric function other than the above functions may be used to associate the first set of features 103 with one or more correlation pairs. The second set of features 108 may be stored in the memory 107 as decision data 207. Referring to the above example, the neural network 102 may be trained to identify a train or objects around the train when there are people in the background. The present disclosure enables the neural network 102 to identify a train or objects around the train when the train is in an open area. Thus, the present disclosure overcomes background variations (e.g., sunlight in an open area) in predictions by the neural network 102 in different environments, thus improving the performance and scalability of the neural network 102.

The other data 208 may store data including temporary data and temporary files generated by one or more modules 202 to perform various functions of the system 104. The other data 208 may be stored in the memory 107. The one or more modules 202 may also include other modules 214 for performing various miscellaneous functions of the system 104. For example, the other module 214 may include a test module. The test module may be configured to generate one or more test features based on the second set of training features and the second set of features 108. Furthermore, the test module may be configured to identify deviations by comparing the one or more test features with the first set of features 103. The test module identifies deviations to determine whether a correct feature (background feature) associated with the first portion of the input data item 101 is predicted. The test module may then be configured to modify the one or more test features in response to identifying the deviations. Furthermore, the test module may be configured to modify the second set of features 108. The test module is used to perform an inverse transformation to modify the second set of features 108. It will be appreciated that one or more of the modules 202 may be represented as a single module or a combination of different modules.

FIG. 4 is an exemplary flowchart illustrating method steps for training a neural network 102 to adapt to a dynamic environment, according to some embodiments of the present disclosure. As shown in FIG. 4, method 400 may be comprised of one or more steps. Method 400 may be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, and functions that perform particular functions or implement particular abstract data types.

The order in which method 400 is described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method. Additionally, individual blocks may be deleted from the method without departing from the scope of the subject matter described herein. Additionally, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

In step 401, the system 104 receives a first set 103 of features of an input data item 101. The input data item 101 may consist of one of an image, a video, an audio input, a text input, and a voice input, and the like. The first set of features 103 is associated with a first portion of the input data item 101. In one embodiment, the system 104 may receive the first set of features 103 from a neural network 102. In another embodiment, the system 104 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like.

In step 402, the system 104 generates one or more first labels and one or more second labels for each data set of the plurality of training data items. The system 104 generates the one or more first labels and the one or more second labels by projecting the first set of training features and the second set of training features onto a feature space. The system 104 projects the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the training data items from the plurality of training data items. Furthermore, the system 104 creates a plurality of regions based on the similarity between the first set of training features and the second set of training features. The system 104 then identifies a center and a radius for each of the plurality of regions. The system 104 adjusts the radius of each of the plurality of regions by projecting the first set of training features and the second set of training features into the feature space for each of one or more classes of the second portion of the other training data items from the plurality of training data items. Additionally, the system 104 generates a label for each of the plurality of regions associated with the corresponding region based on the similarity between the first set of training features and the second set of training features.

In step 403, the system 104 rearranges the first set of training features and the second set of training features in a feature space based on the correlation between the first set of training features and the second set of training features. The system 104 correlates the one or more second labels with the one or more first labels by providing the one or more first labels to the first neural network from the one or more neural networks 102 associated with the generation of the second set of training features. Additionally, the one or more second labels are provided to the second neural network from the one or more neural networks 102 associated with the generation of the first set of training features.

In step 404, the system 104 generates one or more correlation pairs. The system 104 may generate the one or more correlation pairs based on rearrangements of the first set of training features and the second set of training features. Each of the one or more correlation pairs may indicate one or more first labels and corresponding second labels for each of the data sets of the multiple training data items.

FIG. 5 is an exemplary flowchart illustrating method steps for adapting a neural network 102 to a dynamic environment, according to some embodiments of the present disclosure. As shown in FIG. 5, method 500 may be comprised of one or more steps. Method 500 may be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform particular functions or implement particular abstract data types.

The order in which method 500 is described is not intended to be construed as a limitation, and any number of the described method blocks may be combined in any order to implement the method. Additionally, individual blocks may be deleted from the method without departing from the scope of the subject matter described herein. Additionally, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

At step 501, the system 104 receives a first set 103 of features of an input data item 101. The input data item 101 may consist of one of an image, a video, an audio input, a text input, and a voice input, and the like. The first set of features 103 is associated with a first portion of the input data item 101. In one embodiment, the system 104 may receive the first set of features 103 from a neural network 102. In another embodiment, the system 104 may receive the first set of features from one or more sources, such as a database, a user, a capture unit, an audio recorder, and the like.

In step 502, the system 104 determines a second set 108 of features of the input data item 101. The second set 108 of features is associated with a second portion of the input data item 101. The system 104 determines the second set 108 of features by associating the first set 103 of features with one or more correlation pairs. The system 104 may associate the first set 103 of features with one or more correlation pairs by generating a transformation matrix Wt using a Riemannian conditioning technique. A person skilled in the art will understand that any technique other than the techniques described above may be used to associate the first set 103 of features with one or more correlation pairs.

The above-described embodiment and corresponding figures are described by considering the input data items (the plurality of training data items and the input data item 101) as an image. Thus, the first set of features, the second set of features, the first set of training features, the second set of training features, the first label, the second label, etc. correspond to the foreground and background of the image. The present invention is not limited to images, and the first set of features and the second set of features may vary accordingly based on the type of the input data item 101. For example, the input data item 101 may include a speech input for a neural network speech recognition application. In such a case, the first set of features may consist of background features of the speech input, such as the person's accent. The second set of features may consist of foreground features of the speech input, such as the person's voice. In another example, the input data item 101 may include a speech input in a neural network accent classification application. In such a case, the first set of features may consist of background features of the speech input, such as the person's voice. The second set of features may consist of foreground features of the speech input, such as the person's accent.

<Computer System>
6 is a block diagram of an exemplary computer system 600 for implementing embodiments consistent with the present disclosure. In an embodiment, the computer system 600 may be the system 104. Thus, the computer system 600 may be used to adapt the neural network 102 in a dynamic environment. The computer system 600 may include a central processing unit 602 (also referred to as a "CPU" or "processor"). The processor 602 may include at least one data processor. The processor 602 may include specialized processing units such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc. The processor 602 may be used to implement the processor 105 described in FIG. 2.

The processor 602 may be disposed in communication with one or more input/output (I/O) devices (not shown) via an I/O interface 601. The I/O interface 601 may include, but is not limited to, communication protocols/methods such as audio, analog, digital, mono, RCA, stereo, IEEE (Institute of Electrical and Electronics Engineers)-1394, serial bus, universal serial bus (USB), infrared, PS/2, BNC, coaxial, component, composite, digital visual interface (DVI), high definition multimedia interface (HDMI), radio frequency (RF) antenna, S-video, VGA, IEEE 802.n/b/g/n/x, Bluetooth, cellular (e.g., code division multiple access (CDMA), high speed packet access (HSPA+), global system for mobile communications (GSM), long term evolution (LTE), WiMax, etc.), etc.

Using the I/O interface 601, the computer system 600 may communicate with one or more I/O devices. For example, the input device 610 may be an antenna, a keyboard, a mouse, a joystick, an (infrared) remote control, a camera, a card reader, a fax machine, a dongle, a biometric reader, a microphone, a touch screen, a touch pad, a trackball, a stylus, a scanner, a storage device, a transceiver, a video device/source, etc. The output device 611 may be a printer, a fax machine, a video display (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), a plasma, a plasma display panel (PDP), an organic light emitting diode display (OLED), etc.), an audio speaker, etc.

The processor 602 may be disposed in communication with the communication network 609 via the network interface 603. The network interface 603 may communicate with the communication network 609. The network interface 603 may employ a connection protocol including, but not limited to, direct connect, Ethernet (e.g., twisted pair 10/100/1000 base-T), transmission control protocol/internet protocol (TCP/IP), token ring, IEEE 802.11a/b/g/n/x, etc. The communication network 609 may include, but is not limited to, direct interconnect, local area network (LAN), wide area network (WAN), wireless network (e.g., using wireless application protocol), internet, etc. The network interface 603 may employ, but is not limited to, direct connect, Ethernet (e.g., twisted pair 10/100/1000 base-T), transmission control protocol/internet protocol (TCP/IP), token ring, IEEE 802.11a/b/g/n/x, etc. as a connection protocol.

The communication network 609 may include, but is not limited to, a direct interconnection, an electronic commerce network, a peer-to-peer (P2P) network, a local area network (LAN), a wide area network (WAN), a wireless network (e.g., using a wireless application protocol), the Internet, Wi-Fi, and the like. The first and second networks may be dedicated or shared networks, which represent an association of different types of networks that communicate with each other using various protocols, such as, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like. Additionally, the first and second networks may include various network devices, including routers, bridges, servers, computing devices, storage devices, and the like. The computer system 600 may receive data items 612 (the input data items and the plurality of training data items) over the communication network 609.

In some embodiments, the processor 602 may be arranged to communicate with memory 605 (e.g., RAM, ROM, etc., not shown in FIG. 6) via a storage interface 604. The storage interface 604 may connect to memory 605, including, but not limited to, memory drives, removable disk drives, etc., employing connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fibre Channel, Small Computer System Interface (SCSI), etc. The memory drives may further include drum, magnetic disk drives, magneto-optical drives, optical drives, Redundant Array of Independent Discs (RAID), solid state memory devices, solid state drives, etc.

The memory 605 may store a collection of program or database components, including, but not limited to, a user interface 606, an operating system 607, a web browser 608, and the like. In some embodiments, the computer system 600 may store user/application data, such as data, variables, records, and the like, as described in this disclosure. Such databases may be implemented as fault-tolerant, relational, scalable, secure databases, such as Oracle® or Sybase®. The memory 605 may be used to implement the memory 107 described in FIG. 2. The memory 605 may be communicatively coupled to the processor 602. The memory 605 may store instructions executable by one or more processors 602, and when executed, may enable the processor 602 to adapt the neural network 102 to a dynamic environment.

Operating system 607 can facilitate resource management and operation of computer system 600. Examples of operating systems include, but are not limited to, APPLE MACINTOSH R OS X, UNIXR, UNIX-like system distributions (e.g., BERKELEY SOFTWARE DISTRIBUTIONTM (BSD), FREEBSDTM, NETBSDTM, OPENBSDTM, etc.), LINUX DISTRIBUTIONSTM (e.g., RED HATTM, UBUNTUTM, KUBUNTUTM, etc.), IBMTM OS/2, MICROSOFTTM WINDOWSTM (XPTM, VISTATM /7/8, 10, etc.), APPLER IOSTM, GOOGLER ANDROIDTM, BLACKBERRYR OS, etc.

In some embodiments, computer system 600 may implement program components stored in a web browser 608. Web browser 608 may be, for example, a hypertext browsing application such as MICROSOFT INTERNET EXPLORERTM, GOOGLER CHROMETM0, MOZILLAR FIREFOXTM, APPLER SAFARITM, or the like. Secure web browsing is provided using Secure Hypertext Transport Protocol (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), or the like. Web browser 608 may utilize facilities such as AJAXTM, DHTMLTM, ADOBER FLASHTM, JAVASCRIPTTM, JAVATM, application programming interfaces (APIs), and the like. In some embodiments, computer system 600 may implement program components stored in a mail server (not shown). The mail server may be an Internet mail server such as Microsoft Exchange. The mail server may utilize facilities such as ASP™, ACTIVEXT™, ANSI™ C++/C#, MICROSOFT®, .NET™, CGI SCRIPTS™, JAVA™, JAVASCRIPT™, PERLTM, PHP™, PYTHON™, WEBOBJECTS™, etc. The mail server may utilize communications protocols such as Internet Message Access Protocol (IMAP), Messaging Application Programming Interface (MAPI), MICROSOFT® exchange, Post Office Protocol (POP), Simple Mail Transfer Protocol (SMTP), etc. In some embodiments, computer system 600 may implement a mail client stored program component. The mail client (not shown) may be a mail viewing application such as APPLER MAIL™, MICROSOFT® ENTOURAGE™, MICROSOFT® OUTLOOK™, MOZILLAR THUNDERBIRD™, etc.

Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory in which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible objects and exclude carrier waves and transient signals, i.e., non-transient. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, compact disk read-only memory (CD ROM), digital video disk (DVD), flash drives, disks, and any other known physical storage medium.

The present disclosure provides a method and system for adapting a neural network to different dynamic environments, where the neural network is trained by providing a first set of training features (e.g., background features) and a second set of training features (e.g., foreground features) of a plurality of training data items.
The neural network correlates the first set of features and the second set of features to form a correlation pair with a first label (e.g., background label) and a second label (e.g., foreground label). When the neural network receives the first set of features of the input data items (e.g., background features), the neural network determines the second set of features of the input data items (e.g., foreground features) based on the correlation pair. Thus, the network can adapt to new environments. Thus, the present disclosure allows the neural network to adapt to different environments. The present disclosure can be used to develop neural networks under data scarcity constraints for new environments (e.g., new Internet of Things (IoT) environments), thereby expanding the applications of neural networks.

The terms "embodiment", "embodiment", "embodiment", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the invention(s)", unless expressly specified otherwise.

The terms "including," "consisting of," "having," and variations thereof mean "including, but not limited to," unless expressly stated otherwise.

The enumeration of items does not imply that any or all of the items are mutually exclusive unless expressly stated otherwise. The terms "a," "an," and "the" mean "one or more" unless expressly stated otherwise.

The description of an embodiment having multiple components in communication with one another does not imply that all such components are required. Rather, various optional components are described to illustrate the wide variety of possible embodiments of the present invention.

Where a single device or article is described herein, it will be readily apparent that multiple devices/articles (whether they cooperate or not) may be used in place of the single device/article. Similarly, where two or more devices or articles are described herein (whether they cooperate or not), it will be readily apparent that a single device/article may be used in place of the two or more devices or articles, or that a different number of devices/articles may be used in place of the number of devices or programs shown. The functionality and/or features of a device may alternatively be embodied by one or more other devices not expressly described as having such functionality/features. Thus, other embodiments of the present invention need not include a device per se.

The illustrated operations of Figures 4 and 5 show certain events occurring in a particular order. In alternative embodiments, certain operations may be performed in a different order, modified, or removed. Furthermore, steps may be added to the logic described above and still be consistent with the described embodiments. Furthermore, operations described herein may be performed sequentially or certain operations may be processed in parallel. Furthermore, operations may be performed by a single processing unit or by distributed processing units.

Finally, the language used herein has been selected primarily for purposes of readability and explanation, and may not have been selected to define or encompass inventive subject matter. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but rather by the claims that issue on an application based hereon. Accordingly, the disclosure of embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined in the following claims.

While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those of skill in the art. The various aspects and embodiments disclosed herein are intended to be illustrative and not limiting, with the true scope being indicated by the following claims.

100... environment, 101... input data item, 102... neural network, 103... first set of features, 104... system, 105... processor, 106... I/O interface, 107... memory, 108... second set of features, 200... detailed view, 201... data, 202... module, 203... input data, 204... label data, 205... rearrangement data, 206... correlation data, 207... decision data, 208... other data, 209... input module, 210... label generation module, 211... rearrangement module, 212... correlation module, 213... decision module, 214... other modules, 300...example, 301...first part, 302...second part, 303...transformation, 304...second set of features, 305...first set of features, 307...domain, 308...feature space, 309...curved hyperplane, 310...transformation matrix, 600...computer system, 601...I/O interface, 602...processor, 603...network interface, 604...storage interface, 605...memory, 606...user interface, 607...operating system, 608...web browser, 609...communication network, 610...input device, 611...output device, 612...data item

Claims (14)

1. A method for adapting a neural network to a dynamic environment, comprising:
receiving, by the system, a first set of characteristics of the input data item associated with a first portion of the input data item;
determining, by the system, a second set of features of the input data item associated with a second portion of the input data item by associating the first set of features with one or more correlation pairs generated using a neural network;
receiving a feature set from the one or more neural networks, the feature set including a first set of training features and a second set of training features for each data set of a plurality of training data items;
generating, for each data set of the plurality of training data items, one or more first labels and one or more second labels by projecting the first set of training features and the second set of training features onto a feature space;
reordering the first set of training features and the second set of training features in a feature space based on a correlation between the first set of training features and the second set of training features;
generating one or more correlation pairs, each indicative of one or more first labels and corresponding second labels for a respective data set of the plurality of training data items, based on a rearrangement of the first set of training features and the second set of training features;
Including,
method. 10. The method of claim 1 ,
generating one or more test features based on the second set of training features and the second set of features;
identifying deviations by comparing the one or more test features to the first set of features;
modifying the second set of characteristics in response to the deviation; and
Including,
method. 10. The method of claim 1 ,
the input data item and the plurality of training data items include one of an image, a video, an audio input, a text input, and a voice input;
method. 10. The method of claim 1 ,
each dataset of the plurality of training data items comprises an original data item and one or more transformations of the original data item;
method. 10. The method of claim 1 ,
Generating the one or more first labels and the one or more second labels includes:
projecting, from the plurality of training data items, the first set of training features and the second set of training features in the feature space for each of one or more classes of a second portion of the training data items;
creating a plurality of regions based on similarity between the first set of training features and the second set of training features;
identifying a center and a radius for each of the plurality of regions;
adjusting a radius of each of the plurality of regions by projecting the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the other training data items from the plurality of training data items;
generating, for each of the plurality of regions, a label based on a similarity between the first set of training features and the second set of training features associated with a corresponding region;
Including,
method. 6. The method of claim 5,
the number of regions is based on a number of one or more transformations of each of the plurality of training data items and a number of the one or more classes.
method. 10. The method of claim 1 ,
Correlating the one or more second labels with the one or more first labels includes providing the one or more first labels to a first neural network from one or more neural networks associated with generating a second set of training features;
providing the one or more second labels to a second neural network from one or more neural networks associated with generating the first set of training features;
Including,
method. 1. A system for adapting a neural network to a dynamic environment, comprising:
one or more processors;
a memory storing processor-executable instructions that, when executed, cause one or more processors to perform the following operations:
Equipped with
The processor,
Receiving a first set of characteristics of an input data item;
determining a second set of features of the input data items by correlating the first set of features with one or more correlation pairs generated using a neural network;
receiving a feature set from the one or more neural networks, the feature set including a first set of training features and a second set of training features for each data set of a plurality of training data items;
generating, for each data set of the plurality of training data items, one or more first labels and one or more second labels by projecting the first set of training features and the second set of training features onto a feature space;
repositioning the first set of training features and the second set of training features in a feature space based on a correlation between the first set of training features and the second set of training features;
generating one or more correlation pairs indicative of one or more first labels and corresponding second labels for each data set of the plurality of training data items based on a rearrangement of the first set of training features and the second set of training features;
To carry out
system. 9. The system of claim 8,
The one or more processors:
generating one or more test features based on the second set of training features and the second set of features;
identifying deviations by comparing the one or more test features to the first set of features;
modifying the second set of characteristics in response to the deviation; and
To carry out
system. 9. The system of claim 8,
The input data item and the plurality of training data items comprise one of an image, a video, an audio input, a text input, and a voice input. 9. The system of claim 8,
each dataset of the plurality of training data items comprises an original data item and one or more transformations of the original data item;
system. 9. The system of claim 8,
The one or more processors:
projecting, from the plurality of training data items, the first set of training features and the second set of training features in the feature space for each of one or more classes of a second portion of the training data items;
creating a plurality of regions based on similarity between the first set of training features and the second set of training features;
identifying a center and a radius for each of the plurality of regions;
adjusting a radius of each of the plurality of regions by projecting the first set of training features and the second set of training features in the feature space for each of one or more classes of the second portion of the other training data items from the plurality of training data items;
generating, for each of the plurality of regions, a label based on a similarity between the first set of training features and the second set of training features associated with a corresponding region;
Due to
generating the one or more first labels and the one or more second labels;
system. 13. The system of claim 12,
The number of the plurality of regions is based on a number of one or more transformations of each of the plurality of training data items and a number of the one or more classes. 9. The system of claim 8,
The one or more processors:
providing the one or more first labels from one or more neural networks associated with generating the second set of training features to a first neural network, and providing the one or more second labels from the one or more neural networks associated with generating the first set of training features to a second neural network, thereby correlating the one or more first labels with the one or more second labels.
system.

Images