JP7403739B2 - Decision-making device and method for controlling the decision-making device - Google Patents

Description

The present invention relates to a decision-making device and a method of controlling the decision-making device.

Various proposals have been made regarding the Multi-Armed Bandit problem (MAB) as a representative example of a problem that searches for a solution that maximizes the amount of reward obtained stochastically. The bandit problem is a hypothetical problem expressed using a slot machine as an example of a device that provides rewards stochastically. In a system where the player is given a reward with a predetermined probability based on the selected action after repeating the process (whether to play or not), the player is given a reward in the shortest possible time by determining which slot machine to use to maximize the reward. It is a matter of time to decide.

JP 2017-130024 Publication Japanese Patent Application Publication No. 2018-124790

S.-J. Kim, M. Aono, & M. Hara (2010): “Tug-of-war model for the two-bandit problem: Nonlocally-correlated parallel exploration via resource conservation”, BioSystems 101 29-36. S.-J. Kim, M. Aono & E. Nameda (2014): “Efficient decision-making by volume-conserving physical object”, New J. Phys. 17(2015)08302. M. Naruse, M. Berthel, A. Drezet, S. Huant, M. Aono, H. Hori, S.-J. Kim (2015): "Single-photon decision maker", Scientific Reports 5, 13253. M. Naruse, M. Berthel, A. Drezet, S. Huant, H. Hori, & S.-J. Kim (2016): "Single photon in hierarchical architecture for physical reinforcement learning: Photon intelligence", ACS Photonics 2016, 3, 2505-2514. S.-J. Kim, T. Tsuruoka, T. Hasegawa, M. Aono, K. Terabe, & M. Aono, (2016): "Decision maker based on atomic switches", AIMS Materials Science 3(1): 245 -259. C.Lutz, T.Hasegawa, & T.Chikyow (2016): “Ag2S atomic switch-based ‘tug of war’ for decision making”, Nanoscale, 2016, 8, 14031-14036. Takashi Tsuchiya, Tohru Tsuruoka, Song-Ju Kim, Kazuya Terabe, Masakazu Aono (2018): “Ionic decision-maker created as novel, solid-state devices”, Sci. Adv. 2018;4: eaau2057.

Such bandit problems are applied in various fields, such as presenting web advertisements (finding out which advertisements are effective to present). The bandit problem involves determining which option has the maximum reward probability from among N slot machines (options) with binary rewards (“yes” or “absent”) (hereinafter referred to as “binary”). The bandit problem (for example, see Non-Patent Literature 1 and Non-Patent Literature 2), and which option has the maximum expected reward amount from among N slot machines (options) whose rewards are continuous values (real numbers). (hereinafter referred to as the continuous-valued bandit problem; see, for example, Non-Patent Document 4), as well as bandit problems that represent situations in which competition occurs when the number of players is two or more (hereinafter referred to as the competitive binary bandit problem). , for example, see Patent Document 5).

Conventionally, the ε-greedy algorithm, the SOFTMAX algorithm, and the like have been known as methods for searching for solutions to various bandit problems. In recent years, a solution method called the Tug-of-War (TOW) model has also been proposed.

This TOW model is a dynamic algorithm inspired by the behavior of single-celled amoeba-like organisms, and the decision-making function is derived from the physical characteristics of multiple terminals competing for intracellular resources with a finite volume like a tug-of-war. This is what you get. Due to the volume conservation law, which is a physical constraint on the dynamics of this tug-of-war phenomenon, the operation of the TOW model shows higher performance than other well-known algorithms such as the ε-greedy algorithm and the SOFTMAX algorithm. Its high performance has actually been confirmed by conventional algorithms and simulations (Non-Patent Document 1).

Specifically, Fig. 5, it can be seen that the same accuracy rate can be obtained with fewer trials than when using the ε-greedy algorithm or the SOFTMAX algorithm.

Also, Fig. 7 shows an example in which the reward probability distribution for each slot machine (choice) is changed around 3000 trials. Specifically, for slot machines A and B, (Probability distribution of _A = 0 An example is shown in which the probability distribution P _B =0.60) of B is changed to (P _A =0.60, P _B =0.40).

When the reward probability distribution changes in this way, the situation until the correct answer is obtained again is investigated as shown in Fig. of Non-Patent Document 1. As shown in Figure 7, when using the TOW model, the slope of the graph until obtaining the correct answer is steeper than when using the SOFTMAX algorithm, and it can respond to changes in the reward probability distribution more quickly. It is understood that it is more adaptable to environmental changes.

Examples of implementing such a TOW model include a TOW model that utilizes the dynamics of a tug-of-war phenomenon due to physical constraints such as the law of conservation of volume (hereinafter referred to as a dynamic TOW model; see, for example, Non-Patent Document 1); , a TOW model that assumes a virtual bar that does not expand or contract and changes the bar by a fixed value to the left and right (hereinafter referred to as a fixed value bar TOW model, see e.g. Non-Patent Document 2), and a fixed value bar TOW model that changes a fixed value bar TOW model to a non-fixed value. There is a TOW model (hereinafter referred to as a non-fixed value bar TOW model, see, for example, Non-Patent Document 5 and Patent Document 3) that extends to fluctuations.

Furthermore, in recent years, proposals have been made for decision-making devices that physically implement such a TOW model. For example, a method of implementing a TOW model using the particle nature and wave nature of a single photon (for example, see Non-Patent Document 3 and Non-Patent Document 4), or changing the resistance value by applying a pulse voltage, etc. Techniques have been proposed to implement the TOW model using devices that can.

The latter example combines various metals and uses an analog resistance change element whose resistance value changes depending on the applied voltage.The applied voltage is controlled according to the learning data provided to the learning model. The TOW model is implemented by applying the controlled voltage to an analog resistance change element having predetermined characteristics, which serves as a learning model.

According to this, it becomes possible to construct a decision-making device using relatively small hardware without installing software on a large computer or the like.

Note that such an analog resistance change element is an element using an atomic switch (hereinafter referred to as a gapless element) that utilizes the movement, precipitation, and reionization of metal ions in a solid electrolyte sandwiched between electrodes by the application of an electric field. (hereinafter referred to as a gap-type atomic switch), an element that grows or contracts a filament by sandwiching a solid electrolyte between electrodes and applying a voltage (hereinafter referred to as a gap-type atomic switch, for example, Non-Patent Document 6) and electrolyte elements in which an electrolyte material layer capable of transporting ions by an electric field is sandwiched between two or more electrodes (see, for example, Patent Document 2 and Non-Patent Document 7).

When applying decision-making devices using analog resistance change elements such as those described in Patent Documents 1 and 2 and Non-Patent Documents 5, 6, and 7 to the bandit problem, the number of slot machines (choices) is ``2''. A TOW model that solves the bandit problem can be implemented using only one analog resistance change element described in each of the above-mentioned documents.

However, when using the gapless atomic switch described in Patent Document 1 and Non-Patent Document 5 as an analog resistance change element, although it is relatively small and operates with low power consumption, there is a problem that processing speed cannot be increased. There is also no established method for solving large-scale problems that expand the number of slot machines (choices).

Further, a specific method for solving the bandit problem by using a gap type atomic switch as disclosed in Non-Patent Document 6 as an analog resistance change element has not yet been shown. Furthermore, even with an electrolyte element in which an electrolyte material layer capable of transporting ions by an electric field is sandwiched between two or more electrodes, as described in Patent Document 2 and Non-Patent Document 7, the comparison is similar to the gapless atomic switch. Although it is compact and operates with low power consumption, it has the problem of not being able to increase speed, and no method has been found to solve large-scale problems that expand the number of slot machines (options).

As described above, in the conventional technology, it is difficult to increase the number of slot machines (choices) to four or more.

Note that the method of implementing the TOW model using the particle and wave properties of single photons, as described in Non-Patent Document 3 and Non-Patent Document 4, operates at relatively high speed, but does not require miniaturization or It is difficult to reduce power consumption, and attempting to expand the number of slot machines (selections) would significantly increase the size of the device, making it impractical.

The present invention was made in view of the above circumstances, and can be easily miniaturized, operates with relatively low power consumption, and operates at relatively high speed, and is applicable even when there are many options. One of the purposes is to provide a decision-making device and its control method.

One aspect of the present invention that solves the problems of the conventional example is a decision-making device that selects one option from among 2 ⁿ options (n is a natural number), each of which can receive a stochastic reward, A virtual binary tree is constructed using 2 ⁿ -1 circuit elements that can control the value of a predetermined physical quantity with stochastic fluctuations in the positive or negative direction with respect to a reference value, and the lowest layer 2 a circuit unit in which ^n-1 circuit elements are subjected to a process of selecting one of the options; a trial means for selecting one of the options and determining whether or not there is a reward; physical quantity control means for controlling the value of each of the physical quantities of the circuit elements related to the option selected by the trial means according to a predetermined rule according to the result of the judgment; and each circuit element configured in the virtual binary tree. are sequentially selected from the highest circuit element, and depending on whether the value of the physical quantity of the selected circuit element is more positive than the reference value, one of the pair of lower circuit elements is selected, option selection means for selecting one of the options depending on whether the value of the physical quantity of the selected lowest circuit element is more positive than the reference value; The value of the physical quantity of each of the circuit elements related to the option is predetermined so as to increase the probability that the option selected by the trial means is selected by the option selection means when a reward is received by selecting the means. It was decided that the system would be controlled according to established rules.

According to the present invention, it is possible to downsize the decision-making device and operate it with relatively low power consumption. Furthermore, a decision-making device is provided that can operate at relatively high speed and is applicable even when there are many options.

1 is a diagram illustrating a configuration example of a decision-making device according to an embodiment of the present invention. FIG. 3 is a diagram for explaining a multi-armed bandit problem in an embodiment of the present invention. FIG. 3 is a functional block diagram of a decision-making control unit according to an embodiment of the present invention. FIG. 2 is a diagram for explaining a TOW model according to an embodiment of the present invention. 1 is a diagram showing a cross-sectional image obtained by a transmission electron microscope of an analog resistance change element according to an embodiment of the present invention. 1 is a diagram showing a RAND circuit including an analog resistance change element according to an embodiment of the present invention. FIG. 3 is a diagram showing resistance value change characteristics due to application of a pulse voltage of an analog resistance change element used in an embodiment of the present invention. FIG. 3 is a diagram showing a nonlinear resistance value change characteristic due to application of a pulse voltage of an analog resistance change element used in an embodiment of the present invention. FIG. 3 is a diagram for explaining a TOW model for a two-armed bandit problem according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a process when the selected side receives a reward in the TOW model for the two-armed bandit problem according to an embodiment of the present invention. FIG. 6 is a diagram illustrating processing when the selected side does not receive a reward in the TOW model for the two-armed bandit problem according to an embodiment of the present invention. FIG. 2 is a diagram for explaining a TOW model for a multi-armed bandit problem according to an embodiment of the present invention. 2 is a flowchart of an example of processing of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention. FIG. 3 is a diagram illustrating another example of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention. 2 is a flowchart of processing another example of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of performance evaluation of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention. FIG. 7 is a diagram illustrating another example of performance evaluation of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention. FIG. 1 is a configuration block diagram of an example of a decision-making device according to an embodiment of the present invention. FIG. 1 is a functional block diagram of an example of a decision-making device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of a hierarchical structure of a virtual binary tree used by the decision-making device according to the embodiment of the present invention. It is a flowchart figure showing an example of operation of a decision making device concerning an embodiment of the present invention. It is a flowchart figure showing another example of operation of the decision making device concerning an embodiment of the present invention. FIG. 2 is an explanatory diagram showing a configuration example of a circuit of an analog resistance change element of a decision-making device according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a decision-making method and an apparatus thereof according to the present invention will be described with reference to the drawings. Note that even in different drawings, the same reference numerals are used to indicate the same processing and configuration.

(System configuration)
The operation and processing of the decision-making device used in one embodiment of the present invention will be described below. FIG. 1 is a block diagram of an entire decision-making device according to an embodiment of the present invention. This device includes a decision-making control unit 101 that performs central decision-making processing, and a probabilistic reward giving means such as a slot machine that can obtain a maximum reward by changing the resistance value and changing the reward. A RAND controller that determines a necessary voltage based on the analog resistance variable element (RAND) circuit 102 and its input and output, applies it to the analog resistance variable element, controls the analog resistance variable element to obtain a desired resistance value, and performs decision-making processing. 103 and an external interface unit 104 that provides various problems that actually occur to the decision-making control unit 101 as decisionable problems and applies the obtained solutions to the actual problems.

The decision-making control unit 101 inputs and outputs data on the data line 111 to and from the RAND circuit 102, and finally receives a solution. The RAND controller 103 controls the RAND circuit 102 through a control line 121 and inputs/outputs data through a data line 112 to update the RAND. In this embodiment, the processing is executed using the device configuration and the division of functions as described above, but the processing of this embodiment is not limited to this, and the processing of the present embodiment can also be implemented using a configuration and division of functions known in the technical field.

Further, when the external interface unit 104 obtains a problem to be decided or is provided with an optimization problem, the external interface unit 104 converts the problem into the method of the present embodiment using any method known in the technical field. This converts the system into a method that allows decision-making using RAND, and in the future it may be possible to install it in mobile terminals and robots.

FIG. 2 is a diagram for explaining the multi-armed bandit problem according to an embodiment of the present invention. As shown in FIG. 2, the multi-armed bandit problem generally uses more than two slot machines A201, B202, C203, and D204, and the reward probability of each slot machine (for example, P _A = 0.7, P _B = 0.5, P _C = 0.3, P _D = 0.1, etc.), a predetermined amount of coins 205 is used in each slot machine at an arbitrary ratio. The aim is to maximize the reward. For example, if you know that the reward probability (also called winning rate, winning probability, etc.) of slot machine A is the maximum at P _A = 0.7, then if you bet a coin only on slot machine A, you will get the maximum reward, but each slot If the reward probability of a machine is unknown, you can bet coins on all slot machines and proceed with repeated play while estimating the reward probability from the results, so you will have to put some coins into slot machines with low reward probabilities. First, even if it is determined that, for example, slot machine A has the highest reward probability, the overall reward will be lower than when only the slot machine with the highest reward probability is used.

Various algorithms for searching the multi-armed bandit problem have been studied in the past, but the sum of the rewards of the algorithm was often used to evaluate the search methods. However, in recent years, it has become preferable to use the (average) correct answer rate as an evaluation index for decision-making methods (see Non-Patent Document 1). This is because in recent years, in fields such as robots and AI where decision-making methods such as this embodiment are expected to be applied, the environment in which they are used cannot be expected to be in a steady state like a laboratory. This is because in a natural environment, it is important to make decisions as early as possible if it is possible to make more accurate judgments than a certain level. Therefore, the accuracy rate is also used in evaluating the performance using the decision-making system of this embodiment.

FIG. 3 is a functional block diagram of a decision-making control unit according to an embodiment of the present invention. The decision-making control unit 101 is an external interface management module that controls the overall decision-making of this embodiment, communicates with the external interface unit 104, acquires problems in a manner that allows decision-making based on the TOW model, and returns the results. 301 and a resistance value management module 302 that mainly controls the RAND controller 103 to update the resistance value of the RAND circuit 102, which is an analog resistance change element called a Resistive Analog Neuromorphic Device (RAND). Furthermore, after or while changing the resistance value, predetermined data is input to the RAND circuit 102 via the data line 111 to cause the RAND circuit 102 to execute it, and an execution management module 303 is provided for receiving the result. In this embodiment, decision-making processing is performed using such data lines and control, but the decision-making process is not limited thereto, and can be implemented using any data interface or control method known in the technical field.

(TOW model of this embodiment)
FIG. 4 is a diagram for explaining a TOW model according to an embodiment of the present invention. In recent years, the TOW model has been shown to be effective for solving multi-armed bandit problems, especially two-armed bandit problems (see, for example, Non-Patent Document 1). In the embodiment, a virtual bar 401 as shown in FIG. 4 is assumed and used to realize a TOW model.

As shown in FIG. 4, when a slot machine receives a reward, the bar 401 is moved a predetermined distance (for example, 1) toward the slot machine that received the reward (for example, slot machine A411), and the bar 401 is moved in the opposite direction as shown in the bar 402. In this case, the opponent slot machine (for example, slot machine B412) is moved by a predetermined distance (for example, ω). The value of ω is preferably set as ω = ( _PA + P _B )/(2-P _A - P _B ), where the reward probabilities of slot machines _A and B are P A and P _B , respectively. You may take an appropriate value for .

Using such a virtual bar model, a slot machine, which is a stochastic reward giving means, that can ultimately yield the maximum reward is determined. Here, the probabilistic reward giving means is means such as a device that outputs a result of whether or not a reward can be outputted with a fixed reward probability in one process, and corresponds to, for example, a slot machine. That is, for each play, the slot machine outputs the result of either being able to obtain coins (with reward) or not (with no reward) at a predetermined reward probability. Further, by adopting a TOW model using such a virtual bar, it is possible to simultaneously update the evaluation of reward probabilities of two slot machines.

If the TOW model of this embodiment as described above can be implemented in a predetermined hardware element, it is possible to realize a decision-making system that solves a large-scale bandit problem with a device that is simpler, smaller, and consumes less power than conventional devices. Elements with various electrical characteristics are being researched, but if we assume an element whose resistance value changes depending on the applied voltage and set the position of the virtual bar mentioned above as the resistance value, the element's resistance value will change. Due to the characteristics, it was possible to obtain an element in which the behavior of the TOW model of the present embodiment described above corresponds well to the operation of the element. Specifically, depending on the application conditions such as the intensity and pulse width of the pulse voltage to be applied, it has a certain linear characteristic, that is, the amount of change in resistance value with respect to the value of the applied voltage is within a certain range, and it is possible to change the application conditions. By using an analog resistance change element (RAND) that has constant nonlinear characteristics and operates at high speed without hysteresis characteristics, and by associating the position of the virtual bar with the resistance value of RAND, for example, each virtual bar Just like determining which slot machine to play based on the positional relationship with the slot machine, the current resistance value determines which slot machine to play, and if a reward is obtained as a result of playing the slot machine, the resistance value is changed. The value corresponds to the distance to move the virtual bar in the direction determined when playing the same slot machine. Specifically, such a change in resistance value is achieved by applying a voltage for changing the resistance value. By applying a voltage to change the resistance value, if no reward is obtained, to a value corresponding to the distance to move the virtual bar in the direction determined to similarly play the other slot machine. The TOW model of this embodiment can be implemented.

Generally, RAND has an advantage in that it can be adjusted so that the resistance changes linearly under practical pulse application conditions by controlling the laminated structure. This linear resistance change is suitable for faithfully implementing the TOW model. On the other hand, real-world applications often require the ability to flexibly adapt to situations where reward probabilities change dynamically. This means that decisions that were correct before the change in reward probability must be more likely to be withdrawn after the change in reward probability. To this end, it is desirable that the resistance value, which has swung in the direction representing correct decision-making before the reward probability change, can be returned to R _θ as quickly as possible by applying a small number of pulses after the reward probability change. That is, a nonlinear (saturated) response characteristic is also required, in which the resistance change with respect to the pulse voltage becomes smaller as the resistance value increases (or decreases) and swings in one direction. In the RAND of this embodiment, it is also possible to realize such a saturated response characteristic by changing the application conditions such as the intensity and pulse width of the pulse voltage. This allows the reinforcement learning capability to be flexibly optimized depending on the application.

Although such RAND has characteristics of linearity and nonlinearity depending on the pulse voltage application conditions, the element generally has fluctuations 420 with respect to the applied voltage as shown in FIG. There is. Due to this fluctuation 420, a temporary determination is made as to whether or not there is a reward, and based on that determination, a further voltage is applied to change the resistance value. At this time, it is also possible to correspond to a predetermined reward probability by making a difference in the amount of change in the resistance value depending on whether there is a reward or not. In other words, if the voltage that decreases the resistance value is the set voltage, and the voltage that increases (restores) the resistance value is the reset voltage, then the set voltage and the reset voltage are set at a predetermined ratio, for example, ω as shown in FIG. By varying the values, it is possible to repeatedly play and determine an appropriate probabilistic reward granting means.

(RAND structure)
As a RAND having the above characteristics, an element having the following structure could be obtained. FIG. 5 is a diagram showing a cross-sectional image taken by a transmission electron microscope of an analog resistance change element according to an embodiment of the present invention. FIG. 6 is a diagram showing a circuit section including the analog resistance change element of this embodiment.

The RAND of this embodiment has a laminated structure in which a variable resistance layer made of oxide is sandwiched between electrodes made of nitride, and TiN, TiON, TaN, TaON, etc. can be used as the electrodes. Furthermore, as the variable resistance layer made of oxide, not only TaOx but also any material known in this technical field can be used as long as it can stably maintain different oxidation states, such as HfOx, TiOx, and AlOx. can do.

In this embodiment, as shown in FIG. 5, the RAND has a laminated structure and a micropore structure in which TaO _x having different resistance values are sandwiched between TiN electrodes. That is, it has a layer structure in which TaO _x (1) 503 and TaO _x (2) 504 having different resistance values are sandwiched between an upper TiN electrode (TE) 501 and a lower TiN electrode (BE) 502. . Here, TaO _x (1) 503 and TaO _x (2) 504 have different resistance values, and TaO _x (1) 503 has a lower resistance value than TaO _x (2) 504.

Furthermore, as shown in FIG. 5, in order to form a pore structure with a diameter of 100 nm, _two layers of SiO are deposited on the TiN electrode (BE) 502 by chemical vapor deposition (CVD) and then by electron beam lithography. 505 is formed into a film. As can be seen with reference to FIG. 5, the TiN electrode (BE) is completely exposed at the bottom of this micropore structure. On the TiN electrode (BE) exposed by reactive sputtering, layers of TaO _x (2), TaO _x (1) and TiN electrode (TE) are placed and patterned by photolithography and ion milling. Finally, to suppress electrical damage to the device due to transient currents, a 3 kΩ load resistor 602 is adjusted in series with RAND 601, as shown in FIG.

The RAND formed as described above can change its resistance value as shown in FIG. 7 when a set voltage and a reset voltage are applied in a plurality of pulses. At this time, the resistance value of RAND changes linearly with respect to the number of pulse voltage applications without showing hysteresis characteristics.In other words, the amount of change in resistance value due to the pulse voltage is almost the same regardless of the resistance value state. Can be set to .

Specifically, as shown in FIG. 7, the resistance value can be changed at a constant rate by applying a set voltage 701 that decreases the resistance value and a reset voltage 702 that increases (restores) the resistance value a predetermined number of times. can. For example, referring to FIG. 7, when a pulse voltage 702 of -2.0V with a width of 200 ns is applied 50 times as a reset voltage, the resistance value of RAND increases linearly from about 8000Ω to about 9000Ω. On the other hand, when a pulse voltage 701 of +1.7V with a width of 200 ns is applied 50 times as a set voltage, the resistance value of RAND decreases linearly from about 9000Ω to about 8000Ω.

Although it differs each time depending on the set voltage or reset voltage, the resistance value changes by about 20Ω on average, and the amount of change is not affected by the resistance value when applying the reset pulse voltage 50 times. , and even if a set pulse voltage is repeatedly applied 50 times, basically almost the same characteristics can be exhibited.

Further, the amount of change in resistance value with respect to each applied pulse voltage is accompanied by a minute change each time, and has a certain stochastic fluctuation. In the TOW model, these stochastic fluctuations lead to rapid detection of changes in the reward probability distribution in situations where the distribution changes dynamically, and improve the ability to flexibly adapt to the situation. To contribute. Therefore, the RAND of this embodiment is an element suitable for implementing the TOW model in that its resistance value responds linearly to the application of a pulse voltage and changes with stochastic fluctuations. That is understood. Note that in this embodiment, TiN is used as the electrode and TaOx is used as the variable resistance layer made of oxide, but when using other materials, RAND can be fabricated by a method known in the technical field.

On the other hand, in RAND, by changing the application conditions such as the intensity and pulse width of the pulse voltage, as shown in FIG. In a range that is large or small below a certain level, it can be set so that the change in resistance value with respect to the pulse voltage becomes smaller. FIG. 8 is a diagram showing a nonlinear resistance value change characteristic due to application of a pulse voltage of an analog resistance change element used in an embodiment of the present invention. These nonlinear characteristics, as shown in Figure 8, allow the saturated resistance value to quickly return to its reference value by applying a small number of pulses, and provide the ability to flexibly adapt to situations where the reward probability changes dynamically. Contributes to improving

The RAND used in this embodiment and the method for manufacturing RAND have been described above, but the present invention is not limited to this, and any element can be used as long as it has the characteristics required in this embodiment. It can be manufactured using any of the manufacturing methods for analog resistance change elements and other elements known in the technical field.

(Two-armed bandit problem of this embodiment)
An example of processing for making a decision regarding the two-armed bandit problem using the decision-making device incorporating the RAND of this embodiment described above will be described below. FIG. 9 is a diagram for explaining the TOW model for the two-armed bandit problem of this embodiment. FIG. 10 is a diagram showing the process when the selected side gets a reward in the TOW model for the two-armed bandit problem of this embodiment, and FIG. 11 is a diagram showing the process when the selected side does not get the reward. It is a figure which shows a process.

The two-armed bandit algorithm of the TOW model of this embodiment is as follows.
(1 ₎ If R<R _θ (R is on the left side of R _θ in Figures 9 to 11), play slot machine _A. (on the right), play slot machine B (2) If a coin comes out (with reward), the pulse voltage to increase the resistance value by +R ₊₁ in the forward direction (direction in which it is determined that the same slot machine will be played next) If a coin does not come out (no reward), apply a pulse voltage so that the resistance value is -R _ω in the forward direction (the direction in which it is determined that the same slot machine will be played next time) (therefore, it moves in the opposite direction) (3) Add "fluctuation" to the resistance value corresponding to the applied pulse voltage.

Specifically, in this embodiment, a TOW model in which a virtual bar 903 located between slot machine A 901 and slot machine B 902 is moved by slot machine rewards is associated with the RAND resistance value.

That is, first, the slot machine to play is determined based on whether R is larger or smaller than the reference R _θ in association with the position of the bar 903 that determines which slot machine to play.

Looking at FIG. 9, since R<R _θ , it is determined that the slot machine A901 will play, and based on the play result, a set voltage or a reset voltage is applied, and the changed resistance value is obtained. Similar processing is performed on the obtained resistance value, and if the resistance value finally reaches a predetermined value or less, a decision is made to select the slot machine A901. Conversely, when the resistance value reaches a predetermined value or higher, a decision is made to select slot machine B902.

To explain more specifically the process of the two-armed bandit of this embodiment, when it is determined that the slot machine A901 will play in the state shown in FIG. A voltage pulse is applied so that the slot machine A901 changes in the direction in which it will play next time, that is, in the direction in which the resistance value decreases.

More specifically, as shown in FIG. 10, a predetermined set voltage is applied, and as a result, the resistance value changes by R ₊₁ in the direction of the slot machine A901, which changes in the direction of decreasing the resistance value. At this time, since fluctuation 401 is included, although the resistance value is within a certain range, the result will not be the same every play. This makes it possible to search for solutions appropriately.

If it is determined that slot machine A 901 will play in the state shown in FIG. 9, and no coins come out as a result of the play, it is assumed that no reward was obtained, and the direction in which slot machine B 902 plays, that is, the resistance value is A pulse voltage is applied so as to change in an increasing direction. Specifically, as shown in FIG. 11, a predetermined reset voltage is applied, resulting in a change in the direction of slot machine B902 by R _{2 -ω} , which changes in the direction of increasing the resistance value. At this time, since fluctuation 420 is included, although the resistance value is within a certain range, the result will not be the same every play. This allows for an appropriate solution search. The slot machine to play next is determined based on the value of R resulting from this, but in the state shown in FIG. 11, R exceeds R _θ and R>R _θ , so it is determined that slot machine B902 is to be played next. and the above is repeated.

As explained above, in the TOW model using RAND corresponding to the two-armed bandit problem of this embodiment, the resistance value R changes as the play progresses, but the resistance value R(t) for the tth play is as follows. become that way.
R(t)-R _θ =E _B (t)-E _A (t)+δ (1)

Here, E _k (t) (k∈{A, B}) is the evaluation value of each slot machine, and using the number of wins W _k (t) and the number of negative numbers L _k (t) as follows, It is expressed as δ is fluctuation including system noise and system error.
E _A (t) = W _A (t) - ωL _A (t) (2)
E _B (t) = W _B (t) - ωL _B (t) (3)

Here, ω is a control parameter, and the optimal control parameter ω _o is determined as follows.
ω _o =γ/(2-γ) (4)
γ=P _A + P _B (5)

By using only the resistance value behavior in the TOW model of this embodiment, it is possible to obtain a higher accuracy rate than the conventional AI algorithm. The optimal amount of fluctuation is determined by the difference in reward probabilities, but regardless of the fluctuation, a relatively higher accuracy rate is expected.

(Multi-armed bandit of this embodiment)
The two-armed bandit of this embodiment has been described above. In actual applications, there are no two probabilistic reward giving means, so it is necessary to deal with more than two MAB problems. However, for example, even if we try to extend the two-armed bandit processing described above to four terminals, it is extremely difficult to apply the tug-of-war model to four terminals in an actual electronic device. The device fabrication process cannot be considered for industrial use. Therefore, when the number increases beyond four, implementation becomes even more difficult. Therefore, in this embodiment, the ^2n- armed bandit problem is dealt with using two types of hierarchical TOW models described below.

(First example of multi-armed bandit of this embodiment)
FIG. 12 is a diagram illustrating an example of a hierarchical TOW model for a multi-armed bandit problem according to an embodiment of the present invention, and FIG. 13 is a flowchart of an example of processing of the hierarchical TOW model. The MAB of this embodiment uses a plurality of single RANDs, but by providing a hierarchical structure, the two-arm model can be easily extended. That is, as shown in FIG. 12, two RANDs are arranged below the RAND in the upper layer, and which one to execute is selected based on the state of the upper layer and the state of the two lower RANDs. RAND is executed and the resistance value is updated, and the resistance value of RAND in the upper layer is updated.

Although FIG. 12 shows an example of a four-armed bandit, the ^2n- armed bandit problem can be easily solved by applying the above processing to more layers. In the explanation of this embodiment, we will mainly explain how to combine the processes of each RAND using the hierarchical structure, so please refer to the above-mentioned response to the two-armed bandit problem for details such as the process of updating the resistance value of each RAND. About.

To explain the process in more detail with reference to FIG. 13, first, based on the positions of bar 1 in the upper hierarchy, bar 2 and bar 3 in the lower hierarchy, the RAND of bar 2 in the lower hierarchy is determined according to the following table 1. It is determined whether to execute or RAND of bar 3 (step 1301). Referring to Table 1, it can be understood that when bar 1 is to the left of the reference position, one of the slot machines of bar 2 is selected based on the position of bar 2.

Therefore, if the bar 2 is on the left of the reference position, slot machine A is selected regardless of the bar 3's position, and if bar 2 is on the right, slot machine B is selected regardless of the bar 3's position. For example, when RAND of bar 2 in the lower hierarchy is selected and machine A connected here executes a play, the play of slot machine A is executed (step 1302), and this is based on the play result of the selected slot machine. The resistance value of RAND is updated (step 1303). The above process is performed T times (step 1304), and the resistance value of the upper layer is updated based on the result (step 1305).

As described above, by the method of this embodiment, a hierarchical TOW model that solves the ^2n- armed bandit problem can be realized by hierarchically combining single RANDs. Here, in this embodiment, the processing of the lower layer is executed T times, but although the method of this embodiment can be solved well in the case of a slot machine with a relatively easy reward probability distribution, This is because in the case of reward distribution, it is sometimes difficult to solve.

Specifically, let the probabilities of slot machine A, slot machine B, slot machine C _, and slot machine _D be _PA , _PB , PC, _PD , and ( PA , _PB , PC, _{PD )} . In the case of = ( 0.7 , 0.5, 0.3, 0.1), P _A + P _B > P _C + P _D without repeating T times, so the result of the lower layer is directly transferred to the upper layer. Even if the processing is performed in the above manner, it is possible to correctly determine the slot machine A having the highest _PA . On the other hand, in the case of (P _A , P _B , P _C , P _D ) = (0.7, 0.5, 0.9 , 0.1), in this case also P _A + P _B > P _C + P _D Therefore, there is a possibility that bar 2 will be selected and slot machine C, which is the correct answer, will not be selected. For this reason, the update of the upper layer is suspended until the lower layer performs the update process T times and all of them are resolved. Therefore, when the method of this embodiment is used, it will take T times of lower layer processing.

As described above, in this embodiment, the position of bar 1 is determined by comparing two totals: the sum of evaluation values obtained from bar 2 and the sum of evaluation values obtained from bar 3. That is, in this embodiment, R changes by repeating the above process, and the slot machine with the maximum reward is determined based on the value of R. However, the resistance value R(t) after playing t times of bar 1 becomes as follows.
R (t) - R _θ = (E _C (t) + E _D (t)) - (E _A (t) + E _B (t)) + δ (6)
E _k (t)=W _k (t)−ω _(AB) L _k (t) where (k∈{A, B}) (7)
E _k (t)=W _k (t)−ω _(CD) L _k (t) where (k∈{C, D}) (8)

Here, ω _(AB) is a control parameter when kε{A, B}, and ω _(CD) is a control parameter when kε{C, D}.

(Second example of multi-armed bandit of this embodiment)
FIG. 14 is a diagram showing another example of the hierarchical TOW model for the multi-armed bandit problem according to an embodiment of the present invention, and FIG. 15 is a flowchart of processing of another example of the hierarchical TOW model. be. The MAB of this embodiment uses a plurality of single RANDs, but similarly to the first embodiment described above, by providing a hierarchical structure, the two-arm model can be easily expanded. That is, as shown in FIG. 14, two RANDs are arranged below the RAND in the upper layer, and which one to execute is selected based on the state of the upper layer and the state of the two lower RANDs. RAND is executed and the resistance value is updated, and the resistance value of RAND in the upper layer is updated. Although FIG. 14 shows an example of a four-armed bandit, the ^2n- armed bandit problem can be easily solved by applying the above processing to more layers. In this embodiment, the update of the upper layer is performed by performing the TOW model between the slot machines that have obtained the most rewards among the slot machines at bar 2 and bar 3, as in a tournament method, as a result of the TOW model at the lower layer. determines the position of the upper layer bar. As a result, the correct answer can be obtained even in the case of the difficult-to-solve reward distribution described above, without suspending the processing of the upper layer until the processing of the lower layer is repeated T times.

To explain the process in more detail with reference to FIG. 15, first, a slot machine to be played is selected based on the positions of the upper layer and lower layer bars, as in the first embodiment (step 1501). Here, for example, when a selected slot machine in a lower hierarchy executes a play (step 1502), the resistance value of this RAND is updated based on the play result of the selected slot machine (step 1503). Since the RAND of bar 2 is connected to slot machines A and B, and the RAND of bar 3 is connected to slot machines C and D, the slot that has the highest evaluation value among the slot machines A and B of bar 2 The position of bar 1, that is, the resistance value, is determined and updated based on the TOW model of the machine and the slot machine with the highest evaluation value among slot machines C and D of bar 3 (step 1504).

As described above, in this embodiment, the position of bar 1 is determined not by comparing the sum of evaluation values as in the first embodiment described above, but by determining the position of the slot machine with the maximum evaluation value among slot machines A and B of bar 2. It is determined by the TOW model of the slot machine with the highest evaluation value among the slot machines C and D of bar 3. That is, the resistance value R(t) of bar 1 in this example is as follows.
R (t) - R _θ = MAX (E _C (t), E _D (t)) - MAX (E _A (t), E _B (t)) + δ (9)
E _k (t)=W _k (t)−ω _(AB) L _k (t) where (k∈{A, B}) (10)
E _k (t)=W _k (t)−ω _(CD) L _k (t) where (k∈{C, D}) (11)

Here, MAX is a MAX function, which outputs the largest one among the arguments. Therefore, more rewards are output.

As described above, by the method of this embodiment, it is possible to realize a hierarchical TOW model that solves the ^2n- armed bandit problem by hierarchically combining single RANDs. The operating method assumes only a hierarchical TOW model, so it can be used for any TOW model realized using various devices other than devices using RAND. It can also be used in systems using ordinary computers without any special devices.

FIG. 16 is a diagram showing an example of performance evaluation of the hierarchical TOW model for the multi-armed bandit problem according to an embodiment of the present invention, and FIG. 17 is a diagram showing another example of performance evaluation of the hierarchical TOW model. It is a diagram. Here, since the multi-branch model mentioned above, which is difficult to implement, cannot be measured in practice, simulation was used. FIG. 16 shows the performance evaluation when the reward probability distribution (P _A , P _B , P _C , P _D )=(0.7, 0.5, 0.3, 0.1), and FIG. It is a graph showing performance evaluation in the case of reward probability distribution (P _A , P _B , P _C , P _D )=(0.7, 0.5, 0.9, 0.1). Referring to FIGS. 16 and 17, it can be seen that the performance of Example 1 and Example 2 is almost equivalent to that of the branch model.

According to an embodiment of the present invention, each of the 2 ⁿ −1 TOW models implements a TOW model using RAND that can change the response characteristic so that the resistance value changes linearly or nonlinearly with respect to the number of pulse voltage applications. It is possible to integrate individual RANDs on a large scale and operate them hierarchically, and to improve the adaptability to MAB where the reward probability changes ^dynamically . It becomes possible to solve larger-scale decision-making problems than ever before in a compact size, with low power consumption, and at high speed in real time.

Furthermore, by using the RAND of the present invention, the two-armed bandit problem can be executed in a smaller size, with lower power consumption, and faster in real time than before.

In addition, the TOW model enables new and improved hierarchical operational methods to solve the ²ⁿ -armed bandit problem, allowing effective use of the characteristics of various devices implementing the TOW model. Can be done.

[Another explanation]
The decision making device 1 according to the embodiment of the present invention can also be explained as follows. FIG. 18 is a configuration block diagram of an example of the decision-making device 1 according to the embodiment of the present invention.

As illustrated in FIG. 18, the decision-making device 1 according to an example of the present embodiment includes a control section 11, a storage section 12, an analog resistance change element circuit section 13, an input section 14, and an output section 15. It is composed of:

The control unit 11 is configured by a program control device such as a CPU, and operates according to a program stored in the storage unit 12. In this embodiment, the control unit 11 performs a process of selecting one of a plurality of options. Here, each of the options is a means of stochastically providing a reward (some kind of profit), such as multiple types of advertisements placed on a web page or multiple channels available for communication. Specifically, if it is an advertisement, the effect of the advertisement (such as whether it was clicked or not) corresponds to the reward, and if it is a channel for communication, the communication speed may be used as information corresponding to the reward.

The control unit 11 controls the analog resistance variable element circuit unit 13 to select this option, and determines the resistance value of each analog resistance variable element included in the analog resistance variable element circuit unit 13 as a result of the control. Execute by obtaining information. The details of the specific processing performed by the control unit 11 will be explained later.

The storage unit 12 holds programs executed by the control unit 11. This storage section 12 also operates as a work memory for the control section 11. The input unit 14 is a keyboard, a mouse, or the like, and receives a user's instruction operation and outputs the contents of the instruction operation to the control unit 11. The output unit 15 is a display or the like, and presents information to the user according to instructions input from the control unit 11.

The analog resistance change element circuit section 13 includes, for example, at least one RAND (Resistive Analog Neuro Device). Here, as illustrated in FIG. 5, RAND includes a lower electrode (BE) layer 502, an insulating layer 505 having a micropore structure formed with a partial opening on this lower electrode layer 502, and this insulating This is a memory element in which a variable resistance layer 500 that contacts a lower electrode layer 502 through an opening in a layer 505 and an upper electrode (TE) layer 501 formed on the variable resistance layer 500 are laminated in this order.

Here, the variable resistance layer 500 includes a lower layer 504 and an upper layer 503 having different resistance values. Specifically, the upper layer 503 is made of Ta ₂ O _{5 -δ} , the lower layer ₅₀₄ is made of TaO to cause an oxidation or reduction reaction to form a TaO _x channel in a part of this upper layer 503, or return the upper layer 503 to Ta ₂ O _5-δ to narrow (or eliminate) this channel. This makes it possible to change the resistance value.

This change in resistance value can be achieved by applying a positive pulse (a pulsed voltage signal such that the upper electrode layer 501 has a higher potential than the lower electrode layer 502 (referred to as a reset voltage)) or by applying a negative pulse (the upper electrode layer This can be generated by applying a pulsed voltage signal such that the layer 501 has a lower potential (referred to as a set voltage) than the lower electrode layer ₅₀₂ . In the example used in 504, the resistance value is increased when a positive pulse is applied, and the resistance value is decreased when a negative pulse is applied. The cell operation of such devices is widely known and will not be further described here.

Note that the upper electrode layer 501 and the lower electrode layer 502 may each be formed of TiN, and the insulating layer 505 may be formed of SiO ₂ by chemical vapor deposition (CVD). Further, the opening (the diameter of the opening may be about 100 nm) can be formed by electron beam lithography. As a method for forming the variable resistance layer 500 on this upper side, a method such as reactive sputtering can be adopted, for example.

As described above, in one example of the present embodiment, the analog resistance variable element circuit section 13 can be configured using RAND having a laminated structure in which a resistance variable layer made of oxide is sandwiched between electrodes made of nitride. Note that as the material for the upper electrode layer 501 and the lower electrode layer 502, TiON, TaN, TaON, etc. can be used in addition to TiN. In addition, as the variable resistance layer 500 using an oxide, other materials can be used, such as not only TaO _x but also HfO _x , TiO _x , AlO _x, etc., as long as they are all stable materials and have different oxidation states. You can also.

Further, a load resistor of approximately 3 kΩ may be connected in series to each RAND in order to suppress electrical damage to the element due to transient current, and a positive pulse or a negative pulse may be applied via this load resistor.

Since the analog resistance change element circuit section 13 according to an example of the present embodiment uses at least one element having such a configuration, a set signal or a reset signal is applied to this element (RAND) in a pulsed manner multiple times. In this case, the resistance value of each element (RAND) can be changed as illustrated in FIG. In FIG. 7, the reset signal applies a reset voltage of -2.0V for a period of 200 ns, and the set signal applies a set voltage of 1.7 V for a period of 200 ns. As illustrated in FIG. 7, RAND is controlled so that its resistance value monotonically increases when a reset signal is applied, and its resistance value monotonically decreases when a set signal is applied.

Moreover, at this time, the resistance value of RAND changes substantially linearly without showing a hysteresis characteristic with respect to the number of pulse applications within a certain resistance value range. In other words, the amount of change in resistance value caused by one pulse is substantially the same regardless of the state of the resistance value.

That is, as shown in FIG. 4, when a pulse voltage 702 of -2.0V with a width of 200 ns is applied 50 times as a reset voltage, the resistance value of RAND increases linearly in the range of about 8000Ω to about 9000Ω. are doing.

On the other hand, when the pulse voltage 701 of +1.7V with a width of 200 ns is applied 50 times as a set voltage, the resistance value of RAND decreases linearly in the range of about 9000Ω to about 8000Ω. Here, the change in resistance value per pulse differs each time, but the resistance value changes by about 20Ω on average, and the amount of change is essentially constant regardless of the resistance value when applied. It has become. In other words, the resistance value of RAND changes linearly according to the number of pulses applied. Therefore, for example, when changing the resistance value of RAND with the target of +ωΩ, the pulse as the reset voltage is set to ω/ω. It may be applied 20 times (for example, 50 times if ω=1000). Similarly, when changing the resistance value of RAND with -ωΩ as the target, pulses as a set voltage may be applied ω/20 times (for example, 50 times if ω=1000).

Further, as described above, the amount of change in the resistance value each time a pulse voltage is applied is not constant, but involves minute fluctuations (stochastic fluctuations) each time. This characteristic provides an advantageous effect in this embodiment.

In addition, in RAND, outside the resistance value range where the resistance value changes linearly by the application of the pulsed signal, the resistance value changes nonlinearly (in some cases, the resistance value saturates) by the application of the pulsed signal. (Fig. 8).

In this embodiment, an example of RAND is shown in which TiN is used as an electrode and TaOx is used as a variable resistance layer made of an oxide. However, the material constituting RAND is not limited to this, and may be any widely known material. It can be obtained using various materials.

In this way, in the example of this embodiment, a fine circuit structure such as RAND can be used, so miniaturization is easy and power consumption can be suppressed. In addition, since the circuit has a simple configuration, it can operate at relatively high speeds compared to configurations that include multi-stage logic circuits, and it is possible to make (learned) optimal selections in virtually real time. .

In addition, even if it is not RAND, the characteristics required in this embodiment (when controlling to increase or decrease a physical quantity such as resistance value, the average change in physical quantity such as resistance value) Any element can be used as long as it has the characteristic that the amount of change in a physical quantity such as resistance value has stochastic fluctuations each time although the quantity is constant. It can be manufactured using any of the methods for manufacturing analog resistance change elements and other elements.

Next, the operation of the control section 11 will be explained. In the present embodiment, the control unit 11 executes the program stored in the storage unit 12 to control the decision-making processing unit 21, the result determination unit 22, and the resistance value control unit, as illustrated in FIG. A functional configuration including 23 and 23 is realized.

Here, the decision-making processing unit 21 refers to the resistance value of RAND provided in the analog resistance change element circuit unit 13, determines whether the resistance value is larger or smaller than a predetermined reference value, and responds to the result of the judgment. output the decision-making results. This resistance value may be referenced by applying a voltage to RAND from a constant voltage power supply, measuring the amount of current flowing through RAND, and dividing the power supply voltage V by the measured amount of current to obtain the resistance value. Note that when there are multiple RANDs, as schematically shown in FIG. If the switch is turned on (connected state) and the other switches are turned off, voltage is applied from a constant voltage power source to the RAND that is turned on and the current flowing through the RAND is measured, then the resistance value of each RAND can be calculated. can be obtained sequentially.

The simplest example of this decision-making processing section 21 is as follows. That is, when the analog resistance change element circuit unit 13 includes one RAND and performs processing for determining which of advertisements A and B to be posted on a web page based on the resistance value of this RAND, the decision-making processing unit 21 If the reference resistance value is greater than the reference value, a web page with advertisement A may be provided, and if not, a web page with advertisement B may be provided.

The result determination unit 22 determines whether or not there was an outcome (remuneration) corresponding to the decision-making result output by the decision-making processing unit 21, and outputs the determination result. Specifically, when the decision-making processing unit 21 decides which advertisement to post as described above, the result determination unit 22 determines whether or not the advertisement is clicked as the presence or absence of a reward. If it is clicked, there is a result, otherwise it is output as no result.

The resistance value control unit 23 controls the RAND resistance value of the analog resistance variable element circuit unit 13 based on the output of the decision-making processing unit 21 and the result information outputted by the result determination unit 22 to perform the learning process. Execute. Specifically, in the above example, the resistance value control unit 23
(1) When advertisement A is selected and there is an output that shows results, apply a set signal to RAND a predetermined number of times to reduce the resistance value (for example, reduce with -r as the target) (advertisement A is selected). ).
(2) If there is an output with no results when advertisement A is selected, apply a reset signal to RAND a predetermined number of times to increase the resistance value (for example, increase with +ω as the target) (advertisement A is selected) decrease the probability).
(3) When advertisement B is selected and there is an output that shows results, apply a reset signal to RAND a predetermined number of times to increase the resistance value (for example, increase with +r as the target) (advertisement B is selected) increase the probability).
(4) If there is an output with no results when advertisement B is selected, apply a set signal to RAND a predetermined number of times to reduce the resistance value (for example, reduce it with -ω as the target) (advertisement B is selected). ).

Note that the number of times the signal is applied in each of (1) to (4) above does not have to be the same; for example, the number of times the signal is applied in (2) is smaller than the number of times the signal is applied in (1). It's okay.

In this example, the resistance value R(t) after t trials is given by the initial resistance value R _θ .
R (t) - R _θ = E _B (t) - E _A (t) + δ
It can be expressed as. Here, E _A (t) and E _B (t) are the sum of target values when A and B are selected during the trial, respectively:
E _A (t)=r・W _A (t)−ω・L _A (t)
E _B (t)=r・W _B (t)−ω・L _B (t)
, and δ represents (the sum of) stochastic fluctuations that are a characteristic of RAND. In addition, W _k (k = A or B) is the number of times k was selected out of the number of trials and it was determined that there was a result, and L k (k = A or B) is the number of times k was selected out of the number of trials. This is the number of times it was determined that there was no result. ω is a control parameter and is determined experimentally. For example, ω is the probability P _k that option k yields an outcome:
_Pk = _Wk / ( _Wk + _Lk )
Using γ=P _A + P _B ,
ω=γ/(2-γ)
It may be set as

Furthermore, in this embodiment, the options for decision making are not limited to only two, A and B, as in the above example, but can also be applied to a case where there are many options. For example, when there are ²ⁿ options such as A, B, C, D..., when making a decision to select one from these, multiple RANDs are used to perform a virtually hierarchical decision-making process. form a circuit. Specifically, this hierarchical decision-making processing circuit uses 2 ⁿ -1 RANDs and is arranged in a virtual binary tree hierarchical structure as illustrated in FIG. Here, each RAND may be electrically arranged in parallel as illustrated in FIG. 18, but as a process of the control unit 11, as illustrated in FIG. The following processing is performed assuming that the configuration is as follows. Note that here, the 2 ^n-1 RANDs in the lowest layer are actually subjected to the process of selecting one of the options.

In other words, in the example of FIG. 20, each of the 2 ^{n -1} RANDs set in the lowest layer (Nth layer) on the terminal side is selected from A, B, C, D, etc. without ^duplication . 2n ^-2 items are set as related to the selection of a pair of options (A, B), (C, D)..., and are virtually arranged directly above this RAND (N-1st layer). The RANDs are set to select one of a pair of RANDs selected without duplication from among the RANDs of the lower layer (Nth layer). For example, one of the RANDs in the N-1 layer (referred to as the first RAND for convenience) is the RAND corresponding to options A and B in the N-th layer (referred to as the second RAND for convenience). , RAND corresponding to options C and D (referred to as the third RAND for convenience). The relationship between the first RAND and the second RAND and the relationship between the first RAND and the third RAND, which are located in layers adjacent to each other, are hereinafter referred to as a hierarchical association relationship. Hereinafter, RAND up to the highest level (first layer) will be set as a similar configuration.

For example, when n=2 (four options), the decision-making processing unit 21 determines that the resistance value R1 of the first RAND, which is set to be at the top level (first layer), is lower than the predetermined reference value. If it is larger, select the lower second RAND which determines the selection of option A or B; otherwise, select the lower third RAND which determines the selection of option C or D.

Then, when selecting option A or B, the decision-making processing unit 21 calculates the resistance value R2 of the second RAND, which is set as being in the lower (second layer, the lowest layer here) in the virtual hierarchy. Option A is selected when this resistance value R2 is larger than a predetermined reference value. Further, option B is selected when the resistance value R2 is not larger than a predetermined reference value.

Similarly, when selecting option C or D, the decision-making processing unit 21 determines the resistance value of the third RAND, which is set as being in the same lower (second layer) as the second RAND as a virtual layer. With reference to R3, option C is selected if this resistance value R3 is larger than a predetermined reference value. Further, option D is selected when the resistance value R3 is not larger than a predetermined reference value.

That is, the selection of option A or B involves the first RAND (the highest RAND that is the root) and the second RAND that is the end. Further, the selection of option C or D involves the first RAND (the highest RAND, which is the root) and the third RAND, which is the terminal. In this way, a series of RANDs from the root to the end related to the selection of a pair of options is referred to herein as a selection chain.

In this example, the resistance value control unit 23 controls the first to third RAND values of the analog resistance variable element circuit unit 13 based on the output of the decision-making processing unit 21 and the outcome information output from the outcome determination unit 22. The resistance values R1 to R3 are controlled.

Specifically, there are several methods for controlling the resistance value of each RAND in the resistance value control section 23 as follows. Each will be explained below.

[First method]
In the first method, learning processing is performed for each RAND in the lowest layer. That is, in the first method, the following process is executed a predetermined number of times for each RAND in the lowest layer.

Here, the control unit 11 selects one of the corresponding options for each RAND. For example, in the case of the second RAND, either option A or B is selected depending on whether the resistance value of the second RAND is greater than a predetermined reference value. This selection is similar to the operation of the decision-making processing section 21 described above.

Then, as a result of the selection, it is checked whether or not there was a result, and the resistance value of the second RAND is controlled depending on the result of the selection and whether or not there was a result. This control is also similar to the operation in the resistance value control section 23. The control unit 11 performs the process related to the second RAND a predetermined number of times (for example, T times). The control unit 11 performs the same process for the third RAND a predetermined number of times.

In other words, in the case of an advertisement, the control unit 11 tries the operation of presenting an advertisement selected from A and B T times, and controls the resistance value of the second RAND according to the result. The control unit 11 also tries the operation of presenting the advertisement selected from C and D T times, and controls the resistance value of the third RAND according to the result.

In this example of the present embodiment, the control unit 11 calculates the sum ΣE _k of the target values of the respective resistance values of the second and third RANDs during these processes.

That is, as already mentioned, by this operation of the control unit 11, for the second RAND (corresponding to options A and B) after T times, the initial resistance value is set as R _θ ,
R (T) - R _θ = E _B (T) - E _A (T) + δ ₂
It can be expressed as, where E _k (T) = W _k (T) - ωL _k (T) (where k = A, B)
However, in addition to controlling this resistance value, the control unit 11 also controls
ΣE _A,B (T)=E _A (T)+E _B (T)
Find and remember. Similarly, the control unit 11
ΣE _C,D (T)=E _A (T)+E _B (T)
Find and remember.

Note that the provision of the page presenting either advertisement A or B and the provision of the page presenting advertisement C or D may be performed in parallel in terms of time.

In this way, the control unit 11 performs learning processing for the higher-order RAND based on the learning processing results obtained through T trials. Specifically, the control unit 11 determines the resistance value R of an upper layer RAND (in this example, the first RAND) that has a hierarchical relationship with the second and third RANDs. Assuming the initial resistance value as R _θ1 ,
R-R _θ1 = ΣE _C,D (T)-ΣE _A,B (T)+δ ₁
(A pulse signal is applied with ΣE _C,D (T) - ΣE _A,B (T) as the target value).

In addition, if there are three or more layers, calculate ΣEk(T) after updating each RAND of the i+1st layer a predetermined number of times (for example, T times) when performing learning processing on the RAND of the i+1st layer. , the resistance value of RAND of the i-th layer may be set in the same manner as in the above example, and the resistance value of RAND of each layer forming the selection chain may be updated.

After this setting is completed (after the learning process is completed), the control unit 11 starts from the highest RAND and determines whether or not the resistance value of the RAND is larger than a predetermined reference value, as described above. Select one of the RANDs in the second layer, refer to the resistance value of the selected RAND in the second layer, and select the RAND in the third layer depending on whether the referenced resistance value is larger than a predetermined reference value. Select one of the RANDs of the i+1th layer by referring to the resistance value of the selected RAND of the i-th layer, and depending on whether the referenced resistance value is larger than a predetermined reference value. This process is repeated, and it is determined which of the options to select depending on whether the resistance value of RAND selected in the lowest layer is greater than a predetermined reference value.

For example, in the above example, if the resistance value of the first RAND at the highest level is larger than a predetermined reference value, the control unit 11 selects the third RAND (corresponding to options C and D) and With reference to the resistance value of the selected third RAND, if the referenced resistance value of the third RAND is greater than a predetermined reference value, option D is selected. Further, if the referenced resistance value of the third RAND is not greater than the predetermined reference value, the control unit 11 selects option C.

[Second method]
In the second method, the control unit 11 selects an option based on the resistance value of each RAND at the time of processing, as illustrated in FIG. 21 (S2101). In other words, the control unit 11 selects one of the RANDs in the second layer from the highest RAND, depending on whether the resistance value of the RAND is larger than a predetermined reference value, and With reference to the resistance value, one of the RANDs of the third layer is selected depending on whether the referenced resistance value is larger than a predetermined reference value...The selected RAND of the i-th layer (hereinafter referred to as the i-th layer) Depending on whether the referenced resistance value is larger than a predetermined reference value, one of the RANDs in the i+1th layer is selected, and so on. Which of the options to select is determined depending on whether the resistance value of the RAND selected in (referred to as the lowest layer attention RAND in the following description) is greater than a predetermined reference value.

Then, the control unit 11 performs a predetermined process based on the determined selection (for example, provides a web page with an advertisement represented by the selected option), and checks whether there is a result (for example, whether a click was made or not). (S2102). Then, the control unit 11 controls the resistance value of the lowest layer attention RAND based on the selected option and whether or not there is a result (S2103). This control is similar to the operation in the resistance value control section 23.

Further, at this time, the control unit 11 performs the above-mentioned E _k (t) = W _k (t) - ωL _k (t) (where k = A, B, C..., or t is time (number of trials)).
is calculated and obtained as an evaluation value for each option.

Then, the control unit 11 controls the resistance value of the N-1st layer of interest RAND, which is higher than the lowest layer of interest RAND of the lowest layer (Nth layer), as follows.

When the control unit 11 calculates the evaluation value of the RAND of the lower layer (i-th layer) at time t, starting from the lowest layer, the control unit 11 determines the resistance value of the RAND of the i-1th layer at time t, which is the option of the RAND. Using each evaluation value of RAND (each option is U, V and X, Y) of the i-th layer,
R (t) - R _θ = MAX [E _X (t), E _Y (t)] - MAX [E _U (t), E _V (t)] + δ
( _{A pulse} signal _is applied with MAX[ _E Here, MAX[x, y] means taking the larger value of x and y.

That is, the control unit 11 updates the resistance value of RAND on the upper layer side as follows every time Ek on the lower layer side is updated (S2104; upper layer resistance value update). In other words, in the above example, if E _k (k=U, V) of the i-th layer RAND that selects options U and V is updated, the control unit 11 controls the hierarchy for the i-th layer RAND. The resistance value of RAND of the i-1th _layer , which _{has a} relationship of , E _V (t-1)] by setting a target value and controlling it.

Furthermore, in the above example, if Ek (k=X, Y) of the i-th layer RAND that selects options X and Y is updated, the control unit 11 performs a hierarchical The resistance value of RAND of the i-1th layer, which has an association relationship, is set as MAX[E _X (t), E _Y (t)] - MAX[E _X (t-1), The target value is set and controlled so as to decrease by E _Y (t-1)].

As a result, at time t, the resistance value of the RAND becomes
R (t) - R _θ = MAX [E _X (t), E _Y (t)] - MAX [E _U (t), E _V (t)] + δ
The target value is set and controlled so that In this example of the present embodiment, the control unit 11 updates the RAND resistance value of each layer forming the selection chain in this way.

As described above, the control unit 11 executes the selection process in processes S2101 and S2102 while performing the learning process in processes S2103 and S2104. In other words, as described above, the control unit 11 selects one of the second layer RANDs from the highest RAND, depending on whether the resistance value of that RAND is larger than a predetermined reference value, and With reference to the resistance value of the selected RAND of the layer, one of the RANDs of the third layer is selected depending on whether the referenced resistance value is larger than a predetermined reference value... By referring to the resistance value of the RAND, one of the RANDs in the i+1th layer is selected depending on whether the referenced resistance value is larger than a predetermined reference value, and so on. Which of the options to select is determined depending on whether the resistance value of RAND is greater than a predetermined reference value.

For example, in the above example, if the resistance value of the first RAND at the highest level is larger than a predetermined reference value, the control unit 11 selects the third RAND (corresponding to options C and D) and With reference to the resistance value of the selected third RAND, if the referenced resistance value of the third RAND is greater than a predetermined reference value, option D is selected. Further, if the referenced resistance value of the third RAND is not greater than the predetermined reference value, the control unit 11 selects option C.

In addition, in this second method of the present embodiment, in parallel with process S2102, between process S2101 and process S2103, the control unit 11 uses the resistance value of every RAND as a criterion for judgment. Processing may be performed to bring the value close to a predetermined reference value. Specifically, when the reference value is _RT , the control unit 11 assumes that the resistance value of each RAND at time t is R(t),
R'(t)=α(R(t)-R _T )+R _T (that is, α times the resistance value)
(Control with α(R(t)-R _T )+R _T as the target value). Note that α is a real number between 0 and 1, and is determined empirically.

[Example of another method (third method)]
In another method, the control unit 11 selects an option based on the resistance value of each RAND at the time of processing, as illustrated in FIG. 22 (S2201). That is, the control unit 11 selects one of the RANDs in the second layer from the highest RAND depending on whether the resistance value of the RAND is larger than a predetermined reference value, and The selected RAND of the i-th layer (hereinafter referred to as the i-th The RAND of the i+1th layer is selected depending on whether the referenced resistance value is larger than a predetermined reference value, and so on. Which of the options to select is determined depending on whether the resistance value of the RAND selected in the lower layer (referred to as the lowest layer attention RAND in the following description) is greater than a predetermined reference value.

Next, the control unit 11 performs a process of bringing the resistance values of all RANDs close to a predetermined reference value that is a criterion for determination (S2202). Specifically, when the reference value is RT, the control unit 11 assumes that the resistance value of each RAND at time t is R(t),
R'(t)=α(R(t)-R _T )+R _T (that is, α times the resistance value)
(Control with α(R(t)-R _T )+R _T as the target value). Note that α is a real number between 0 and 1, and is determined empirically.

On the other hand, the control unit 11 tries the option selected in step S2201, and determines whether or not it was successful (S2203). Specifically, in this process, if the option is an advertisement pasted on a web page, whether or not the advertisement is successful is determined based on whether or not the advertisement is clicked.

The control unit 11 controls the resistance value of the lowest layer RAND corresponding to the option selected in step S2201, according to the result of the determination in step S2203 (S2204). In other words, if the RAND of the lowest layer (Nth layer) corresponds to options X and Y (bottom layer noted RAND) and its resistance value is greater than a predetermined reference value, select option Y; otherwise, select option Y. If option X is to be selected in this case, the control unit 11:
(1) If it is determined that there is a result when option
(2) If it is determined that there is no result when selecting _option
(3) When selecting option Y and determining that there is a result, apply a reset signal to the corresponding RAND a predetermined number of times to increase the resistance value (for example, increase +r as a target value),
(4) If it is determined that there is no result when option Y is selected, a set signal is applied to the corresponding RAND a predetermined number of times to reduce the resistance value (for example, reduce it to −ω _s as a target value).

Furthermore, the control unit 11 controls the RAND (i+1st layer attention RAND) that is in the i+1st layer and selected in the predetermined S2201 among the RANDs in the i-th (i=1, 2,...N-1) layer. For a RAND (i-th layer attention RAND) that has a hierarchical association relationship, a pair of RANDs (one of which is the i+1 layer attention RAND) in a lower layer (i+1 layer) that has a hierarchical association relationship with this i-layer attention RAND. As RANDx, RANDy,
(1) When RANDx is selected and it is determined that there is a result, a set signal is applied a predetermined number of times to the i-layer attention RAND to reduce the resistance value (for example, -r is set as the target value),
(2) When RANDx is selected and it is determined that there is no result, a reset signal is applied a predetermined number of times to the i-th layer attention RAND to increase the resistance value (for example, increase +ω _i as a target value);
(3) When RANDy is selected and it is determined that there is a result, a reset signal is applied to the i-layer attention RAND a predetermined number of times to increase the resistance value (for example, increase +r as a target value);
(4) When RANDy is selected and it is determined that there is no result, a set signal is applied to the i-layer focused RAND a predetermined number of times to reduce the resistance value (for example, -ω _i is set as the target value) (upper layer update Processing; S2205). Note that here, ω _i is a target value for the i-th layer, and a different value may be set for each layer.

The control unit 11 sequentially performs this upper layer update process on each layer from the N-1st layer to the first layer, and updates the resistance value of the RAND of interest (RAND selected in process S2201) in each layer included in the selection chain. Update. The control unit 11 repeatedly executes the processing from S2201 to S2205.

In this example, the selection result in step S2201 can be used as is as a result of the selection process. Note that if it is determined that the learning process has been completed, only process S2201 may be repeatedly executed; however, if the probability of obtaining results for each option varies (for example, in the case of advertising, it may vary depending on the season). By repeating the processes from S2201 to S2205, it is possible to select options (decision making) that follow changes.

Further, in the description up to this point, the analog resistance change element circuit section 13 may use a circuit with a crossbar structure as illustrated in FIG. 23. As shown in FIG. 23, this circuit includes first and second wiring groups 2301 and 2302 that are arranged in different layers and intersect in plan view (in FIG. 23, a wiring group extending in the left-right direction is called a first wiring group). 2301, and a group of wires intersecting perpendicularly thereto is referred to as a second group of wires 2302). In FIG. 23, for convenience of illustration, RAND (or variable resistance element) is shown with the symbol of a normal resistor.

When such an analog resistance variable element circuit unit 13 is used to realize the example of forming a virtually hierarchical decision-making processing circuit according to this embodiment, the number of RANDs in the lowest layer is 2 ⁿ . If so, use a circuit having at least 2 ⁿ × 2 ⁿ wiring groups (intersections).

In this circuit, the intersection between the first row of wires in the first wire group 2301 and the first column wire in the second wire group 2302 (hereinafter referred to as the intersection of the first row and first column) RAND1-1 of 2nd row 1st column and 1st row 2nd column RAND2-1, 1-2 of The RAND at each intersection on the virtual line segments 2303-1, 2303-2... connecting the intersections of i= ^2k , k=1, 2..., n) is used (other RANDs are not used).

In other words, if each RAND on the line segment 2303-k is used as the RAND of the k-th layer in the hierarchical decision-making processing circuit, the decision-making device 1 of this embodiment can be processed by the various methods described above. realizable. However, the present embodiment is not limited to this example, and a different allocation method may be adopted as to which grid point's RAND is allocated as each RAND in the hierarchical decision-making processing circuit.

[Modified example]
In the explanation so far, an analog resistance change element circuit is used, and the resistance value of the circuit is used to function as a selection element that selects one of a pair of options. However, the physical quantity used as a selection criterion is not limited to only the resistance value, but also includes stochastic fluctuations in the positive or negative direction with respect to a predetermined reference value (in other words, it fluctuates stochastically with respect to control over the target value). changes in the relevant direction, and if control is possible such that the average amount of change is significant (requires measurement), then it is an element that can change other physical quantities such as the direction and strength of the magnetic field. There may be.

Further, the example of the decision-making target of the decision-making device 1 according to the present embodiment is not limited to the above-mentioned advertisement example, but may be selected from among a plurality of options, for example, the communication channel. selection (in this case, the reward depends on whether or not there was a communication channel available, that is, whether communication was successful), and the selection of the frequency when extracting electricity from the vibration power generation device (in this example, the reward depends on whether or not there was a free communication channel, that is, whether communication was possible) (depending on whether electricity can be extracted or not).

1 Decision-making device 11 Control section 12 Storage section 13 Analog resistance change element circuit section 14 Input section 15 Output section 21 Decision-making processing section 22 Result judgment section 23 Resistance value control section 101 Decision-making control section 102 RAND circuit 103 RAND controller 104 External Interface section 111, 112 Data line 121 Control line 201, 202, 203, 204 Slot machine 205 Coin 301 External interface management module 302 Resistance value management module 303 Execution management module 401, 402 Bars 411, 412 Slot machine 420 Fluctuation 501 TiN electrode ( T.E.)
502 TiN electrode (BE)
503 TaO _x (1)
504 TaO _x (2)
505 SiO ₂ layer 506 Substrate 601 RAND
602 Load resistance 701 Set voltage operation 702 Reset voltage operation 901, 902 Slot machine 903 Bar

Claims (9)

A decision-making device element that determines one of two or more stochastic reward granting means based on a TOW (Tug-of-War) model,
A circuit means whose resistance value changes by applying a predetermined pulse voltage, wherein the resistance value changes linearly with respect to the number of times the pulse voltage is applied, and further the resistance value has stochastic fluctuations. changing circuit means;
Depending on the magnitude of the current resistance value of the circuit means with respect to a predetermined threshold value, it is determined which of the first stochastic reward giving means and the second stochastic reward giving means is to be executed, and the reward As a result of the granting process, a pulse voltage is applied to change the resistance value in a direction in which it is determined that the stochastic reward granting means that has been executed further executes the reward granting process when there is a reward, and when there is no reward circuit control means for applying a pulse voltage for changing the resistance value in a direction in which it is determined that a stochastic reward giving means different from the stochastic reward giving means that executed the reward giving process executes the reward giving process;
A decision-making device that determines one from three or more probabilistic reward giving means using a decision-making device element comprising :
A hierarchical structure formed by arbitrarily selecting two of all stochastic reward giving means to form a pair to form a lower hierarchy, and further combining each pair to form a pair to form an upper hierarchy. and associating the decision-making device elements with respect to each of the formed pairs, determining one of the stochastic reward granting means according to a set of resistance values of all the decision-making device elements , and determining the stochastic reward granting means. Upper layer processing that executes lower layer processing that updates the resistance value of the corresponding decision making device element according to the reward result of the means, and sequentially updates the resistance value of the higher layer decision making device element based on the updated value. A decision-making device characterized in that by repeating the above, it is determined to execute a reward grant process of one stochastic reward granting means from three or more stochastic reward granting means. The decision-making device according to claim 1 , wherein the upper layer processing is executed after the lower layer processing has been executed a predetermined number of times. 2. The upper layer process is performed so as to select a probabilistic reward giving means with a higher evaluation value based on the result of the executed reward process as a result of the lower layer process. decision-making device. Based on the TOW model (Tug-of-War), in which a solution is obtained by a virtual bar negotiation between two parties, a decision-making method is developed in which a decision is made to decide one of three or more probabilistic reward giving means. A program to be executed by a computer, the decision making method comprising:
a hierarchy forming step of arbitrarily selecting two of all stochastic reward giving means to form a pair to form a lower hierarchy, and further combining each pair to form a pair to form an upper hierarchy;
For each pair of the formed lowest hierarchy, the pair of stochastic reward giving means is arranged on both sides of the virtual bar of the TOW model, and the virtual positional relationship of the virtual bar of the lowest hierarchy is determined. and a lowest reward granting step of determining and executing the reward granting process of which probabilistic reward granting means is to be executed based on the positional relationship of the virtual bars of all the layers above it;
As a result of the process of the lowest level reward granting step , the virtual bar of the lowest level is moved in the direction in which it is determined that the stochastic reward granting means that has executed the reward granting process further executes the reward granting process based on the result that there is a reward. is moved by a predetermined distance, and based on the result of no reward, the virtual of the lowest hierarchy is moved in the direction in which it is determined that a stochastic reward giving means different from the stochastic reward giving means that executed the reward giving process executes the reward giving process. a bottom update step of moving the bar by a predetermined distance;
and an upper update step in which, based on the lowest update step, updates are performed sequentially by reflecting the updated values of the immediately lower hierarchy in the virtual bars of each higher hierarchy. program. A decision-making device that selects one option from among ²ⁿ options (n is a natural number) each of which can receive a reward with probability,
A virtual binary tree is constructed using 2 ⁿ -1 circuit elements that can control the value of a predetermined physical quantity with stochastic fluctuations in the positive or negative direction with respect to a reference value, and the lowest layer 2 a circuit unit in which ^n-1 circuit elements are subjected to a process of selecting one of the options;
trial means for selecting one of the options and determining whether there is a reward;
physical quantity control means for controlling the value of the physical quantity of each of the circuit elements related to the option selected by the trial means according to a determination result of the trial means, according to a predetermined rule;
Each of the circuit elements configured in the virtual binary tree is selected sequentially starting from the highest level circuit element, and depending on whether the value of the physical quantity of the selected circuit element is more positive than the reference value, the pair of lower order option selection means for repeatedly selecting one of the circuit elements and selecting one of the options depending on whether the value of the physical quantity of the selected lowest circuit element is more positive than the reference value; ,
has
The physical quantity control means includes:
In order to increase the probability that the option selected by the trial means is selected by the option selection means when a reward is received due to the selection of the trial means, the value of the physical quantity of each of the circuit elements related to the choice is set. A decision-making device that controls according to predetermined rules. The decision-making device according to claim 5 ,
The physical quantity control means includes:
When a reward is received as a result of the selection of the trial means, among the circuit elements configured in the virtual binary tree, the circuit elements of each layer involved in the selection of the option selected by the trial means are set as circuit elements of interest. , controlling the value of the physical quantity of each circuit element of interest according to a predetermined rule so as to increase the probability that the circuit element of interest in the option or the lower layer is selected, respectively;
If no reward is received due to the selection of the trial means, among the circuit elements configured in the virtual binary tree, the circuit elements of each layer involved in the selection of the option selected by the trial means are designated as circuit elements of interest. A decision-making device that controls the value of the physical quantity of each circuit element of interest according to a predetermined rule so as to reduce the probability that the option or the circuit element of interest in a lower layer is selected. The decision-making device according to claim 5 or 6 ,
The physical quantity control means includes:
A decision-making device that controls the value of the physical quantity of each circuit element to approach a predetermined value before controlling the value of the physical quantity of the circuit element according to the predetermined rule. The decision-making device according to any one of claims 5 to 7 ,
The decision-making device wherein the physical quantity is a resistance value of a circuit, and the circuit element is a variable resistance element. Each can receive rewards with probability 2 ⁿ A method for controlling a decision-making device that selects one option from among (n is a natural number) options, the method comprising:
The decision-making device includes two circuit elements capable of controlling the value of a predetermined physical quantity with stochastic fluctuations in a positive or negative direction with respect to a reference value. ⁿ -1 is used to construct a virtual binary tree, and the bottom 2 ^n-1 subjecting the circuit elements to a process of selecting one of the options,
The trial means selects one of the options and determines whether there is a reward,
A physical quantity control means controls the value of each of the physical quantities of the circuit elements related to the option selected by the trial means according to a predetermined rule, based on the determination result of the trial means,
The option selection means sequentially selects each circuit element configured in the virtual binary tree starting from the highest circuit element, and determines whether the value of the physical quantity of the selected circuit element is more positive than the reference value. , repeating the process of selecting one of the pair of lower-order circuit elements, and selecting one of the options depending on whether the value of the physical quantity of the selected lowest-order circuit element is more positive than the reference value. death,
The physical quantity control means includes:
In order to increase the probability that the option selected by the trial means is selected by the option selection means when a reward is received due to the selection of the trial means, the value of the physical quantity of each of the circuit elements related to the choice is set. A method for controlling a decision-making device according to predetermined rules.

Images