[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2024135137A1 - Imaging device - Google Patents

Imaging device Download PDF

Info

Publication number
WO2024135137A1
WO2024135137A1 PCT/JP2023/040305 JP2023040305W WO2024135137A1 WO 2024135137 A1 WO2024135137 A1 WO 2024135137A1 JP 2023040305 W JP2023040305 W JP 2023040305W WO 2024135137 A1 WO2024135137 A1 WO 2024135137A1
Authority
WO
WIPO (PCT)
Prior art keywords
router
data
routers
light receiving
pixel
Prior art date
Application number
PCT/JP2023/040305
Other languages
French (fr)
Japanese (ja)
Inventor
晋 宝玉
Original Assignee
ソニーセミコンダクタソリューションズ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーセミコンダクタソリューションズ株式会社 filed Critical ソニーセミコンダクタソリューションズ株式会社
Publication of WO2024135137A1 publication Critical patent/WO2024135137A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70SSIS architectures; Circuits associated therewith
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70SSIS architectures; Circuits associated therewith
    • H04N25/76Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/77Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
    • H04N25/772Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising A/D, V/T, V/F, I/T or I/F converters
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70SSIS architectures; Circuits associated therewith
    • H04N25/76Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/779Circuitry for scanning or addressing the pixel array
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70SSIS architectures; Circuits associated therewith
    • H04N25/79Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors

Definitions

  • This disclosure relates to an imaging device that utilizes Spiking Neural Network (SNN) hardware.
  • SNN Spiking Neural Network
  • NNs neural networks
  • SNN spiking neural network
  • SNNs use spike signals as a means of transmitting information, and are a form of NN that allows spike signals to be processed asynchronously by maintaining intermediate states in which individual neurons change according to input spikes. Due to these characteristics, SNNs are said to contribute to improved processing speed and power consumption compared to conventional NNs. Signal processing using SNNs is disclosed, for example, in Patent Document 1.
  • a single-level or multiple-level router is connected to a pixel array section having multiple light-receiving pixels and a multiprocessor section having multiple processors configured by SNN hardware. This reduces congestion in data transmission from the pixel array to the multiprocessor compared to when the pixel array and the multiprocessor are connected via a conventional serializer.
  • FIG. 1 is a diagram illustrating an example of a schematic configuration of an imaging device according to a first embodiment of the present disclosure.
  • FIG. 2 is a diagram illustrating an example of a layered configuration of the imaging device illustrated in FIG.
  • FIG. 3 is a diagram illustrating an example of a functional block of the light receiving pixel illustrated in FIG.
  • FIG. 4 is a diagram illustrating an example of a functional block of the router on the sensor unit side illustrated in FIG.
  • FIG. 5 is a diagram illustrating an example of functional blocks of a router on the processor unit side illustrated in FIG.
  • FIG. 6 is a diagram illustrating an example of functional blocks of the processor illustrated in FIG.
  • FIG. 7 is a block diagram showing a modified example of the stacked structure of the imaging device shown in FIG. FIG.
  • FIG. 8 is a diagram illustrating an example of functional blocks of the router on the light receiving pixel side illustrated in FIG.
  • FIG. 9 is a diagram illustrating an example of a schematic configuration of an imaging device according to the second embodiment of the present disclosure.
  • FIG. 10 is a diagram illustrating an example of a layered configuration of the imaging device illustrated in FIG.
  • FIG. 11 is a diagram illustrating an example of functional blocks of the router illustrated in FIG.
  • FIG. 12 is a diagram illustrating an example of functional blocks of the processor illustrated in FIG.
  • FIG. 13 is a diagram illustrating an example of functional blocks of the pixel array unit illustrated in FIGS.
  • FIG. 14 is a diagram showing an example of time series values of the membrane potential in the membrane potential memory unit shown in FIGS.
  • FIG. 15 is a diagram illustrating an example of transmission data.
  • FIG. 16 is a diagram illustrating an example of transmission data.
  • FIG. 17 is a diagram illustrating an example of functional blocks of the processor unit illustrated in FIG.
  • FIG. 18 is a diagram showing a modified example of the functional blocks of the router shown in FIG.
  • FIG. 19 is a diagram showing a modified example of the functional blocks of the router shown in FIG.
  • FIG. 20 is a diagram showing an example of the transmission prohibition table shown in FIG.
  • FIG. 21 is a diagram showing a modified example of the functional blocks of the router shown in FIG.
  • FIG. 22 is a diagram showing a modified example of the functional blocks of the router shown in FIG. 23A, 23B, 23C, and 23D are schematic diagrams showing whether data output is possible or not, determined based on the control signal ctl3.
  • FIG. 23A, 23B, 23C, and 23D are schematic diagrams showing whether data output is possible or not, determined based on the control signal ctl3.
  • FIG. 23A, 23B, 23C, and 23D
  • FIG. 24 is a diagram showing a modification of the functional blocks of the cell of FIG.
  • FIG. 25 is a diagram showing a modified example of the functional blocks of the cell of FIG.
  • FIG. 26 is a diagram showing an example of the operating state of a plurality of neurons in a processor.
  • FIG. 27 is a diagram illustrating an example of data bypass within a processor.
  • FIG. 28 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. 29A and 29B are diagrams illustrating an example of an output image from the router of the imaging device illustrated in FIG. 28 and an image obtained after the output image is decoded.
  • FIG. 30 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. FIG.
  • FIG. 31 is a diagram illustrating a modification of the schematic configuration of the imaging device shown in FIG.
  • FIG. 32 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG.
  • FIG. 33 is a diagram illustrating a modification of the schematic configuration of the imaging device shown in FIG.
  • FIG. 34 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG.
  • FIG. 1 shows a schematic configuration example of an imaging device 1000 according to an embodiment of the present disclosure.
  • the imaging device 1000 includes a sensor unit 100 and a processor unit 200.
  • the sensor unit 100 includes a pixel array unit 110 and a router 120.
  • the processor unit 200 includes a core array unit 210.
  • the core array unit 210 includes a plurality of cores C arranged two-dimensionally. Each core C includes a processor 211 and a router 212.
  • the core array unit 210 includes a plurality of processors 211 arranged two-dimensionally and a plurality of routers 212 arranged two-dimensionally.
  • the plurality of routers 212 are assigned to each processor 211.
  • the two-dimensionally arranged processors 211 correspond to a specific example of a "multiprocessor" according to an embodiment of the present disclosure.
  • the processor 211 is configured by SNN (Spiking Neural Network) hardware.
  • SNN Spiking Neural Network
  • the pixel array unit 110 and each processor 211 are directly connected via routers 120 and 212 .
  • the pixel array unit 110 has a plurality of light receiving pixels P arranged two-dimensionally.
  • the light receiving pixels P include, for example, a CMOS (Complementary Metal Oxide Semiconductor) element, an EVS (Event-Based Vision Sensor) element, or a SPAD (Single Photon Avalanche Diode) element.
  • the light receiving pixels P generate detection signals by detecting light incident from the outside.
  • the pixel array unit 110 may, for example, directly transmit the generated detection signals to the router 120.
  • the pixel array unit 110 may, for example, digitize the detection signals using an ADC (Analog to Digital Converter) or a counter in the pixel array unit 110, and transmit the resulting digital signals to the router 120.
  • the pixel array unit 110 transmits data (for example, the detection signals or the digital signals) generated based on the light detection at the light receiving pixels P to the router 120 as pixel data Dp.
  • the detection signal or the digital signal corresponds to a so-called spike signal.
  • the pixel array section 110 may transmit pixel data Dp obtained from each light receiving pixel P to the router 120 based on control data ctl2 from the router 120.
  • This control data ctl2 includes, for example, data for each light receiving pixel P regarding whether data output from the pixel array section 110 is required.
  • the pixel array section 110 may determine whether or not it is required to output pixel data Dp obtained from each light receiving pixel P based on the control data ctl2, and transmit the pixel data Dp of the light receiving pixel P determined to require output to the router 120.
  • the router 120 When the router 120 acquires pixel data Dp from the pixel array unit 110, it references the pixel address corresponding to the acquired pixel data Dp to acquire the address of the neuron to which the pixel data Dp is to be sent.
  • the router 120 is provided with a routing table, and the router 120 acquires the address of the neuron to which the pixel data Dp is to be sent based on the routing table.
  • the router 120 may generate time data (timestamp) when the pixel data Dp was acquired, if necessary.
  • the router 120 transmits transmission data DA, which includes the acquired address and pixel data Dp, to each core C.
  • the router 120 may determine the destination of the transmission data DA based on control data ctl1 from the core array unit 210.
  • This control data ctl1 includes, for example, data on the operating state of each neuron in the core array unit 210.
  • the state of a neuron includes, for example, data on whether it is waiting for processing (busy) or not.
  • the router 120 may transmit data about the operating state of the digital converters included in the pixel array unit 110 (for example, the digital converters included in the readout units 113, 116, and 119 described below) to each router 212 in the core array unit 210.
  • This operating state may include, for example, whether the digital converter is operating, the bit width of the digital conversion, and the operating timing or operating frequency of the digital converter.
  • a plurality of routers 212 are arranged two-dimensionally in the core array unit 210.
  • the router 212 determines the destination of the transmission data DA by referring to the address included in the transmission data DA acquired from the router 120.
  • the router 212 is provided with a routing table, and determines the destination of the transmission data DA based on the routing table.
  • the router 212 may determine the destination of the transmission data DA based on the address included in the transmission data DA acquired from the router 120 and data on the operating state (the operating state of the digital converter included in the pixel array unit 110) acquired from the router 120.
  • the router 212 transmits the transmission data DA to the determined destination. If the determined destination is a neuron in the processor 211 corresponding to the router 212, the router 212 transmits the transmission data DA to the processor 211 corresponding to the router 212. If the determined destination is a neuron in the processor 211 corresponding to the router 212 adjacent to the router 212, the router 212 transmits the transmission data DA to the processor 211 corresponding to the router 212 adjacent to the router 212.
  • the processor 211 performs signal processing using SNN on the pixel data included in the transmission data DA acquired from the router 212 corresponding to the processor 211.
  • the core array unit 210 outputs data Dout obtained by signal processing in the processor 211 to the outside.
  • the router 212 generates data about the operating state of the neuron in the processor 211 corresponding to the router 212 based on data (spike signals and addresses) obtained from the LIF unit 211d (described below) in the processor 211 corresponding to the router 212.
  • the router 212 may transmit the generated data (data about the operating state of the neuron in the processor 211 corresponding to the router 212) to the router 120.
  • the sensor chip 1000A is provided with one or more pad electrodes PE1 for each light receiving pixel P. Each pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to wiring through which transmission data DA is transmitted.
  • the SNN chip 1000B is provided with one or more pad electrodes PE2 for each light receiving pixel P. Each pad electrode PE2 is provided on the surface of the SNN chip 1000B, and is connected to the input terminal of the router 120.
  • the sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
  • the input terminal of the router 120 is connected to each light receiving pixel P, and the output terminal of the router 120 is connected to each core C (router 212).
  • the router 120 is provided at a location opposite to the location where multiple routers 212 are provided.
  • the router provided between the pixel array unit 110 and each processor 211 is a multi-layer (two-layer) structure consisting of a first-layer router 120 and multiple second-layer routers 212.
  • Figures 3(A) to 3(C) show examples of the configuration of a light-receiving pixel P.
  • the light receiving pixel P may include a CMOS element.
  • the light receiving pixel P includes, for example, a photoelectric conversion unit 111 and a charge storage unit 112 as shown in FIG. 3A.
  • the photoelectric conversion unit 111 includes, for example, a photodiode, and performs photoelectric conversion on light incident on the light receiving surface of the sensor chip 1000A to generate a charge according to the amount of light received.
  • the charge storage unit 112 includes, for example, a transfer transistor electrically connected to the photodiode, and a floating diffusion that temporarily holds the charge output from the photodiode via the transfer transistor.
  • the charge storage unit 112 outputs, for example, a voltage signal according to the level of the charge held in the charge storage unit 112 as a detection signal.
  • the light receiving pixel P may further include a readout unit 113, for example, as shown in FIG. 3A.
  • the readout unit 113 includes a digital converter that digitally converts a voltage signal (detection signal) corresponding to the level of the charge stored in the charge storage unit 112, and an output circuit that outputs a digital signal (pixel data Dp) obtained by digital conversion.
  • the pixel array unit 110 may further include a row readout circuit that reads raster data including one row of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the read raster data to the router 120.
  • the row readout circuit sequentially outputs the digital signals (pixel data Dp) obtained from the light receiving pixels P for each pixel row.
  • Visible light image data for example, RGB image data
  • the light receiving pixel P may include an EVS element.
  • the light receiving pixel P includes, for example, a photoelectric conversion unit 114 and a subtraction unit 115, as shown in FIG. 3B.
  • the photoelectric conversion unit 114 includes, for example, a photodiode, and performs photoelectric conversion on the light incident on the light receiving surface of the sensor chip 1000A to generate a charge according to the amount of light received.
  • the subtraction unit 115 includes, for example, a buffer and a sample and hold circuit. The buffer holds a voltage signal according to the level of the charge output from the photodiode.
  • the sample and hold circuit samples the signal supplied from the buffer, holds the sampled signal, and then outputs a signal according to the difference between the signal supplied from the buffer and the signal held in the sample and hold circuit as a detection signal.
  • the subtraction unit 115 functions like a memory in the light receiving pixel P.
  • the light receiving pixel P may further include a readout unit 116, for example, as shown in FIG. 3B.
  • the readout unit 116 includes a digital converter that digitally converts the signal (detection signal) output from the subtraction unit 115, and an output circuit that outputs a digital signal (pixel data Dp) obtained by digital conversion.
  • the pixel array unit 110 may further include a row readout circuit that reads out raster data including one row's worth of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the read data to the router 120.
  • the row readout circuit sequentially outputs the digital signals (pixel data Dp) obtained from the light receiving pixels P for each pixel row.
  • EVS image data is generated by the digital signals obtained from the light receiving pixels P.
  • the light receiving pixel P may include a SPAD element.
  • the light receiving pixel P includes, for example, a SPAD section 117 and a pulse detection section 118, as shown in FIG. 3C.
  • the SPAD section 117 includes, for example, a SPAD element, operates in Geiger mode, and generates an avalanche current when a photon is incident in a state where a negative bias voltage equal to or greater than the breakdown voltage is applied between the anode and cathode of the SPAD element.
  • the pulse detection section 118 includes, for example, a quench resistor connected in series to the SPAD section 117, and an inverter connected to a connection node between the SPAD section 117 and the quench resistor.
  • the inverter outputs a high-level signal when the voltage of the connection node is lower than a predetermined threshold voltage (i.e., when it is at a low level).
  • the inverter outputs a low-level signal when the voltage of the connection node is equal to or greater than a predetermined threshold voltage (i.e., when it is at a high level).
  • the detection section 118 functions as a digital converter that outputs a digital signal as a detection signal.
  • the light receiving pixel P may further include a readout unit 119, for example, as shown in FIG. 3C.
  • the readout unit 119 includes a counter that counts the signal output from the pulse detection unit 118 and outputs a signal (pixel data Dp) based on the count value.
  • the pixel array unit 110 may further include a row readout circuit that reads out raster data including one row's worth of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the readout raster data to the router 120.
  • the row readout circuit sequentially outputs multiple digital signals obtained from the multiple light receiving pixels P for each pixel row. Visible light image data (e.g., RGB image data) is generated by the multiple digital signals obtained from the multiple light receiving pixels P.
  • Visible light image data e.g., RGB image data
  • the readout units 113, 116, and 119 or the row readout circuit may receive control data (control data ctl2) from the router 120.
  • the readout units 113, 116, and 119 or the row readout circuit may output pixel data Dp to the router 120 based on the control data (control data ctl2) from the router 120.
  • the pixel array section 110 has a row readout circuit.
  • one or more pad electrodes PE1 may be connected to the row readout circuit.
  • the input terminal of the router 120 is connected to the row readout circuit, and the output terminal of the router 120 is connected to each core C.
  • the digital converter provided in the light-receiving pixel P may be provided within the row readout circuit.
  • FIG. 4 shows an example of a schematic configuration of the router 120.
  • the router 120 has, for example, an input port 121, a FIFO (first-in first-out) memory unit 122, a destination address determination unit 123, an arbiter 124, and an output port 125, as shown in Fig. 4.
  • FIFO first-in first-out
  • the input port 121 is electrically connected to each pad electrode PE2, and outputs a plurality of pixel signals Dp transmitted from the pixel array section 110 to the FIFO memory section 122.
  • the FIFO memory section 122 temporarily stores the plurality of pixel signals Dp input from the input port 121.
  • the FIFO memory section 122 sequentially outputs the plurality of pixel signals Dp stored in the FIFO memory section 122 under the control of the arbiter 124.
  • the destination address determination section 123 has a routing table (TBL), and acquires the address of the neuron to which each pixel data Dp is to be transmitted based on the TBL.
  • the destination address determination section 123 associates the pixel signal Dp with the acquired address and outputs it to the arbiter 124.
  • the arbiter 124 arbitrates requests for output of pixel signals Dp supplied from each of the multiple light receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 125.
  • the arbiter 124 may perform arbitration according to control data ctl1, for example.
  • the arbiter 124 may determine the destination of the transmission data DA based on the control data ctl1, for example.
  • the control data ctl1 includes data on the operating state of each neuron in the core array unit 210, for example.
  • the arbiter 124 may, for example, output the control data ctl2 generated based on the arbitration result to the pixel array unit 110.
  • the arbiter 124 may, for example, generate the control data ctl2 based on the control data ctl1 and output the generated control data ctl2 to the pixel array unit 110.
  • the control data ctl2 includes, for example, data for each light receiving pixel P regarding whether or not data output from the pixel array unit 110 is required.
  • the output port 125 transmits the transmission data DA to each core C.
  • the configuration of the router 120 is not limited to the configuration shown in FIG. 4.
  • the configuration of the router 120 is not limited to the configuration shown in FIG. 4 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
  • FIG. 5 shows an example of a schematic configuration of the router 212.
  • the router 212 has an input port 212a, a FIFO memory unit 212b, a destination address determination unit 212c, an arbiter 212d, and an output port 212e.
  • the input port 212a of the router 212 is connected to the output port 125 of the router 120.
  • the input port 212a of the router 212 is connected to the output port 212e of four adjacent routers 212.
  • the output port 212e of the router 212 is connected to the input port 212a of four adjacent routers 212.
  • the output port 212e of the router 212 is connected to the processor 211 in the common core C.
  • the input port 212a has four ports (eastIN, southIN, westIN, northIN), one port (UpIN), and one port (DownIN).
  • the four ports (eastIN, southIN, westIN, northIN) are connected to the output ports 212e of four adjacent routers 212.
  • One port (UpIN) is connected to the output port 125 of the router 120.
  • One port (DownIN) is connected to the processor 211 in the common core C.
  • the input port 212a outputs the input transmission data DA to the FIFO memory unit 212b.
  • the FIFO memory unit 212b temporarily stores the transmission data DA input from the input port 212a.
  • the FIFO memory unit 212b outputs the transmission data DA stored in the FIFO memory unit 212b according to the control of the arbiter 212d.
  • the destination address determination unit 212c has a routing table (TBL) and obtains the address of the neuron to which the transmission data DA is to be sent based on the TBL.
  • the destination address determination unit 212c associates the obtained address with the pixel signal Dp and outputs it to the arbiter 212d.
  • the arbiter 212d arbitrates requests for output of multiple pieces of transmission data DA input to the input port 212a, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of the transmission data DA) to the output port 212e.
  • the arbiter 212d may, for example, output control data ctl1 generated based on the arbitration result to the router 120.
  • the control data ctl1 includes, for example, the operating state of each neuron in the core array unit 210.
  • the output port 212e outputs the transmission data DA to the input ports 212a of the four adjacent routers 212, the input port 121 of the router 120, or the processor 211 in the common core C.
  • the configuration of the router 212 is not limited to the configuration shown in FIG. 5.
  • the configuration of the router 212 is not limited to the configuration shown in FIG. 5 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
  • processor 211 6 shows an example of a functional block of the processor 211.
  • the processor 211 is configured by SNN hardware.
  • the processor 211 performs signal processing using SNN on pixel data Dp included in the transmission data DA received from the router 212.
  • the processor 211 has a neuron I/O unit 211a, a product-sum operation unit 211b, a weight storage memory unit 211c, a membrane potential memory unit 211d, and an LIF (Leaky integrate-and-fire) unit 21d.
  • the configuration of the processor 211 is not limited to the configuration shown in FIG. 6.
  • the neuron I/O unit 211a outputs pixel data Dp contained in the transmission data DA received from the router 212 as a spike signal to the product-sum calculation unit 211b.
  • the neuron I/O unit 211a outputs the pixel data Dp received from the router 212 to the product-sum calculation unit 211b in association with the neuron destination address received from the router 212.
  • the product-sum calculation unit 211b multiplies the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each neuron destination address, and performs a product-sum calculation to add up the number of input spikes for each neuron destination address.
  • the product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a.
  • the weight value is stored in the weight storage memory unit 211c.
  • the neuron I/O unit 211a stores the value (result of the product-sum operation) for each neuron destination address received from the product-sum operation unit 211b as a membrane potential in the membrane potential memory unit 211d.
  • This membrane potential is what is called an intermediate state.
  • an intermediate state is defined for each neuron, and changes based on the input from the product-sum operation unit 211b via the neuron I/O unit 211a.
  • the LIF unit 211d performs leaky integration and firing processing.
  • the LIF unit 211d multiplies the membrane potential stored in the membrane potential memory unit 211d by a predetermined membrane time constant, thereby causing a temporal change (leakage) in the membrane potential.
  • the LIF unit 211d further outputs a spike signal to the router 212 when one or more values of the intermediate state exceed a predetermined threshold.
  • the LIF unit 211d outputs the spike signal to the router 212 in association with the address of the neuron whose intermediate state exceeds the predetermined threshold.
  • the pixel array unit 110 and each processor 211 are directly connected via routers 120 and 212. This makes it possible to reduce congestion (the destination of spikes concentrating at one location at the same time) in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
  • the router 120 is connected to multiple light receiving pixels P or row readout circuits and multiple processors 211. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN. Note that when a row readout circuit is provided, it is not necessary to wait for readout from all light receiving pixels P when outputting data from the pixel array unit 110. Therefore, it is possible to reduce congestion, compared to when reading out from all light receiving pixels P.
  • the digital signal obtained by the above-mentioned digital converter is transmitted to multiple processors 211 via routers 120 and 212.
  • the routers 120, 212 are provided with FIFO memories 122, 212b and arbiters 124, 212d, and multiple digital signals stored in the FIFO memories 122, 212b are sequentially transmitted to multiple processors 212 under the control of the arbiters 124, 212d.
  • This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to the conventional case where the pixel array and the multiprocessor are connected via a serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
  • a first-level router 120 is provided that is assigned to a plurality of light-receiving pixels P, and a plurality of second-level routers 212 are provided, one for each processor 211.
  • the router 120 is connected to each light receiving pixel P or row readout circuit and each router 212.
  • data on the operating state of the digital converter described above is transmitted by the router 120 to each router 212.
  • each router 212 determines the destination of the digital signal obtained by the digital converter described above based on the data on the operating state obtained from the router 120.
  • congestion in data transmission from the pixel array unit 110 to each processor 211 can be reduced compared to when the pixel array and the multiprocessor are connected via a conventional serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
  • data on the operating state of neurons in processor 211 is transmitted to router 120 by router 212 corresponding to processor 211.
  • router 120 to determine the destination of data obtained from light-receiving pixel P based on the data from router 212.
  • congestion in data transmission from pixel array unit 110 to each processor 211 can be reduced compared to the conventional case where a pixel array and a multiprocessor are connected via a serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
  • the sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other. This makes it possible to shorten the data transmission distance from the pixel array 110 to each processor 211 compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using SNN.
  • the pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to a wiring that outputs the digital signal obtained by the above-mentioned digital converter.
  • the pad electrode PE2 is provided on the surface of the SNN chip 1000B, and is connected to the input end of the router 120.
  • FIG. 7 shows a modified example of the stacked structure of the imaging device 1000 shown in FIG. 2.
  • the router 120 may be configured, for example, by a plurality of routers 120A arranged two-dimensionally as shown in FIG. 7.
  • the multiple light receiving pixels P are divided into multiple groups (first groups), and the multiple light receiving pixels P divided into each first group constitute the pixel array section 110A.
  • the pixel array section 110 is composed of multiple pixel array sections 110A arranged two-dimensionally.
  • the multiple cores C are divided into multiple groups (second groups), and the multiple cores C divided into each second group constitute the core array section 210A.
  • the core array section 210A is composed of multiple cores C arranged two-dimensionally.
  • the routers 120A are assigned to the pixel array units 110A one by one, and further assigned to the core array units 210A one by one.
  • the routers 120A are connected to the corresponding pixel array units 110A (each light receiving pixel P or row readout circuit) and to the corresponding core array units 210A (each router 212).
  • the routers 120A are further connected to the adjacent routers 120A.
  • the routers 120A are provided at locations opposite to the locations where the corresponding pixel array units 110A (multiple light receiving pixels P) are provided, and are provided at locations opposite to the locations where the corresponding core array units 210A (multiple routers 212) are provided.
  • the routers provided between the pixel array unit 110 and each processor 211 are multiple hierarchical (two hierarchical) layers, consisting of multiple routers 120A in the first layer and multiple routers 212 in the second layer.
  • each router 120A is further connected to the output port 125 of the four adjacent routers 120A.
  • the output port 125 of each router 120A is connected to the input port 121 of the four adjacent routers 120A.
  • the sensor chip 1000A is provided with one or more pad electrodes PE1 for each light receiving pixel P. Each pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to a wiring that outputs the transmission data DA.
  • the SNN chip 1000B is provided with one or more pad electrodes PE2 for each light receiving pixel P. Each pad electrode PE2 is provided on the surface of the SNN chip 1000B. Each pad electrode PE2 provided in the pixel array section 110A is connected to the input terminal of the router 120A corresponding to the pixel array section 110A.
  • the sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
  • FIG. 8 shows an example of the schematic configuration of router 120A.
  • router 120A has an input port 121, a FIFO memory unit 122, a destination address determination unit 123, an arbiter 124, and an output port 125.
  • the input port 121 is electrically connected to each corresponding pad electrode PE2, and outputs a plurality of pixel signals Dp transmitted from the corresponding pixel array unit 110A to the FIFO memory unit 122.
  • the input port 121 further outputs a plurality of pixel signals Dp transmitted from the output ports 125 of the four adjacent routers 120A to the FIFO memory unit 122.
  • the FIFO memory unit 122 temporarily stores multiple pixel signals Dp input from the input port 121.
  • the FIFO memory unit 122 sequentially outputs the multiple pixel signals Dp stored in the FIFO memory unit 122 under the control of the arbiter 124.
  • the destination address determination unit 123 has a routing table (TBL) and obtains the address of the neuron to which each piece of pixel data Dp is to be sent based on the TBL.
  • the destination address determination unit 123 associates the pixel signal Dp with the obtained address and outputs it to the arbiter 124.
  • the arbiter 124 arbitrates requests for output of pixel signals Dp supplied from each of the multiple light receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 125.
  • the arbiter 124 may perform arbitration according to control data ctl1, for example.
  • the arbiter 124 may determine the destination of the transmission data DA based on the control data ctl1, for example.
  • the arbiter 124 may, for example, output the control data ctl2 generated based on the arbitration result to the pixel array unit 110.
  • the arbiter 124 may, for example, generate the control data ctl2 based on the control data ctl1 and output the generated control data ctl2 to the pixel array unit 110.
  • the output port 125 transmits the transmission data DA to each core C and to the input port 121 of the adjacent router 120A.
  • the configuration of router 120A is not limited to the configuration shown in FIG. 8. As long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL, the configuration of router 120A is not limited to the configuration shown in FIG. 8.
  • a plurality of routers 120A in the first hierarchical layer are provided, one for each of a set of the first group and the second group, and a plurality of routers 212 in the second hierarchical layer are provided, one for each of a set of the first group and the second group.
  • This allows data transmission for each of a set of the first group and the second group.
  • congestion in data transmission from the pixel array unit 110 to each processor 211 can be reduced compared to when the pixel array unit and the multiprocessor unit are connected via a serializer as in the conventional case. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
  • the router 120A is connected to each light receiving pixel P of the corresponding first group and each router 212 of the corresponding second group, and is further connected to multiple routers 120A adjacent to the corresponding first group.
  • Each router 212 is connected to the corresponding router 120A and the corresponding processor 211, and is further connected to multiple adjacent routers 212.
  • data on the operating state of the digital converter described above is transmitted by the router 120 to each router 212.
  • the destination of the digital signal obtained by the digital converter described above is determined by the router 212. This makes it possible to transmit data according to the degree of congestion of each processor 211. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using an SNN.
  • data on the operating state of neurons in processor 211 is transmitted to router 120A by router 212 corresponding to processor 211.
  • router 120A determines the destination of data obtained from light-receiving pixel P based on the data from router 212.
  • congestion in data transmission from pixel array unit 110 to each processor 211 can be reduced compared to the conventional case where the pixel array and multiprocessor are connected via a serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using SNN.
  • FIG. 9 shows a schematic configuration example of an imaging device 2000 according to a second embodiment of the present disclosure.
  • the imaging device 2000 includes a sensor unit 300 and a processor unit 400.
  • the sensor unit 300 includes a pixel array unit 310.
  • the processor unit 400 includes a core array unit 410.
  • the core array unit 410 includes a plurality of cores C arranged two-dimensionally.
  • Each core C includes a processor 211 and a router 412.
  • the core array unit 410 includes a plurality of processors 211 arranged two-dimensionally and a plurality of routers 412 arranged two-dimensionally.
  • the plurality of processors 211 arranged two-dimensionally correspond to a specific example of a "multiprocessor" according to an embodiment of the present disclosure.
  • the processor 211 is configured by SNN hardware.
  • the pixel array unit 310 and each processor 211 are directly connected via a plurality of routers 412.
  • the pixel array section 310 has a plurality of light receiving pixels P arranged two-dimensionally.
  • the light receiving pixels P include, for example, a CMOS element, an EVS element, or a SPAD element.
  • the light receiving pixels P generate detection signals by detecting light incident from the outside.
  • the pixel array section 310 may, for example, directly transmit the generated detection signals to the plurality of routers 412.
  • the pixel array section 310 may, for example, digitize the detection signals using an ADC or a counter in the pixel array section 110, and transmit the digital signals thus obtained to the plurality of routers 412.
  • the pixel array section 310 transmits data (for example, the detection signals or the digital signals) generated based on the light detection at the light receiving pixels P as pixel data Dp to the plurality of routers 412.
  • the detection signals or the digital signals correspond to so-called spike signals.
  • the pixel array unit 310 may transmit pixel data Dp obtained from each light receiving pixel P to the router 412 based on control data ctl2 from the multiple routers 412.
  • This control data ctl2 includes, for example, data for each light receiving pixel P regarding whether data output from the pixel array unit 310 is required.
  • the pixel array unit 310 may determine whether or not it is required to output the pixel data Dp obtained from each light receiving pixel P based on the control data ctl2, and transmit the pixel data Dp of the light receiving pixel P determined to require output to the router 412.
  • the router 412 When the router 412 acquires pixel data Dp from the pixel array unit 310, it references the pixel address corresponding to the acquired pixel data Dp to acquire the address of the neuron to which the pixel data Dp is to be sent.
  • the router 412 is provided with a routing table, and uses the routing table to acquire the address of the neuron to which the pixel data Dp is to be sent. If necessary, the router 412 may generate time data (timestamp) when the pixel data Dp was acquired.
  • the router 412 transmits transmission data DA including the acquired address and pixel data Dp. If the destination is a neuron in the processor 211 corresponding to the router 412, the router 412 transmits the transmission data DA to the processor 211 corresponding to the router 412. If the destination is a neuron in the processor 211 corresponding to the router 412 adjacent to the router 412, the router 412 transmits the transmission data DA to the processor 211 corresponding to the router 412 adjacent to the router 412.
  • the processor 211 performs signal processing using SNN on the pixel data Dp included in the transmission data DA acquired from the router 412.
  • the core array unit 410 outputs data Dout obtained by signal processing in the processor 211 to the outside.
  • the router 412 determines the destination of the digital signal obtained by the digital converter included in the pixel array unit 110 (for example, the digital converter included in the readout units 113, 116, and 119) based on data about the operating state of the digital converter.
  • This operating state includes, for example, whether the digital converter is operating, the bit width of the digital conversion, and the operating timing or operating frequency of the digital converter.
  • the router 412 generates data on the operating state of the neuron in the processor 211 corresponding to the router 412 based on the data (spike signal and address) obtained from the LIF unit 211d in the processor 211 corresponding to the router 412.
  • the router 412 may generate control data Ctl2 based on the generated data (data on the operating state of the neuron in the processor 211 corresponding to the router 412) and transmit the generated control data Ctl2 to the pixel array 310.
  • FIG. 10 shows an example of a layered structure of the imaging device 2000.
  • the pixel array section 310 is formed, for example, by a sensor chip 2000A as shown in FIG. 10. In the sensor chip 2000A, the pixel array section 310 is formed on a semiconductor substrate.
  • the processor section 400 is formed, for example, by an SNN chip 2000B as shown in FIG. 10. In the SNN chip 2000B, the processor section 400 is formed on a semiconductor substrate.
  • the sensor chip 2000A is provided with one or more pad electrodes PE1 for each light receiving pixel P.
  • Each pad electrode PE1 is provided on the surface of the sensor chip 2000A opposite the light receiving surface, and is connected to wiring that outputs pixel data Dp.
  • the SNN chip 2000B is provided with one or more pad electrodes PE2 for each light receiving pixel P.
  • Each pad electrode PE2 is provided on the surface of the SNN chip 2000B, and is connected to the input terminal of the router 412.
  • the sensor chip 2000A and the SNN chip 2000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
  • the multiple light receiving pixels P are divided into multiple groups (first groups), and the multiple light receiving pixels P divided into each first group constitute the pixel array section 310A.
  • the pixel array section 310 is composed of multiple pixel array sections 310A arranged two-dimensionally.
  • multiple routers 412 are arranged two-dimensionally, and multiple processors 211 are arranged two-dimensionally.
  • the routers 412 are assigned one by one to the pixel array units 310A, and further assigned one by one to the processors 211.
  • the router 412 is connected to the corresponding pixel array unit 310A (each light receiving pixel P) and to the corresponding processor 211.
  • the router 412 is further connected to adjacent routers 412.
  • the router 412 is provided at a location opposite the location where the corresponding pixel array unit 310A (multiple light receiving pixels P) is provided, and is provided at a location opposite the location where the corresponding processor 211 is provided.
  • the router 412 may be provided at a location opposite the location adjacent to the location where the corresponding processor 211 is provided.
  • the router provided between the pixel array unit 310 and each processor 211 is a single layer (one layer) composed of multiple routers 412.
  • the input port 412a of each router 412 is further connected to the output port 412e of the four adjacent routers 412a.
  • the output port 412e of each router 412 is connected to the input port 412a of the four adjacent routers 412.
  • FIG. 11 shows an example of a schematic configuration of the router 412.
  • the router 412 has an input port 412a, a FIFO memory unit 412b, a destination address determination unit 412c, an arbiter 412d, and an output port 412e.
  • the input port 412a of the router 412 is connected to each light receiving pixel P (or row readout circuit) of the corresponding pixel array 310A.
  • the input port 412a of the router 412 is connected to the output port 412e of four adjacent routers 412.
  • the output port 412e of the router 412 is connected to the input port 412a of four adjacent routers 412.
  • the output port 412e of the router 412 is connected to the processor 211 in the common core C.
  • the input port 412a has four ports (eastIN, southIN, westIN, northIN), one port (PxIN), and one port (localIN).
  • the four ports (eastIN, southIN, westIN, northIN) are connected to the output ports 412e of the four adjacent routers 412.
  • One port (PxIN) is connected to the corresponding pixel array unit 310A.
  • One port (localIN) is connected to the processor 211 in the common core C.
  • the input port 412a outputs the input pixel data Dp to the FIFO memory unit 412b.
  • the FIFO memory unit 412b temporarily stores multiple pixel signals Dp input from the input port 412a.
  • the FIFO memory unit 412b outputs the multiple pixel signals Dp stored in the FIFO memory unit 412b according to the control of the arbiter 412d.
  • the destination address determination unit 412c has a routing table (TBL) and obtains the address of the neuron to which each pixel data Dp is to be sent based on the TBL.
  • the destination address determination unit 412c associates the obtained address with the pixel signal Dp and outputs it to the arbiter 412d.
  • the arbiter 412d arbitrates requests for output of pixel signals Dp supplied from each of the multiple light-receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 412e.
  • the arbiter 412d may, for example, output control data ctl1 generated based on the arbitration result to the arbiter 412d of an adjacent router 412.
  • the control data ctl1 includes, for example, the operating state of each neuron in the core array unit 410.
  • the output port 412e outputs the pixel signal Dp to either the input port 412a of the four adjacent routers 412 or the processor 211 in the common core C.
  • the configuration of the router 412 is not limited to the configuration shown in FIG. 11.
  • the configuration of the router 412 is not limited to the configuration shown in FIG. 11 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
  • FIG. 12 shows an example of a functional block of the processor 211.
  • the processor 211 is composed of SNN hardware.
  • the processor 211 performs signal processing using SNN on the pixel signal Dp received from the router 412.
  • the processor 211 has a neuron I/O unit 211a, a product-sum operation unit 211b, a weight storage memory unit 211c, a membrane potential memory unit 211d, and an LIF unit 211d.
  • the configuration of the processor 211 is not limited to the configuration shown in FIG. 12.
  • the neuron I/O unit 211a outputs the pixel signal Dp received from the router 412 to the product-sum calculation unit 211b as a spike signal.
  • the product-sum calculation unit 211b performs a product-sum calculation by multiplying the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each neuron destination address and adding up the number of input spikes for each neuron destination address.
  • the product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a.
  • the above weight values are stored in the weight storage memory unit 211c.
  • the neuron I/O unit 211a receives the value (result of the product-sum operation) for each neuron destination address from the product-sum operation unit 211b and stores it as a membrane potential in the membrane potential memory unit 211d.
  • This membrane potential is what is called an intermediate state.
  • the LIF unit 211d performs leaky integration and firing processing.
  • the LIF unit 211d multiplies the membrane potential stored in the membrane potential memory unit 211d by a predetermined membrane time constant, thereby causing a temporal change (leakage) in the membrane potential.
  • the LIF unit 211d further outputs a spike signal to the router 412 when one or more values of the intermediate state exceed a predetermined threshold.
  • one router 412 is assigned to each processor 211.
  • the pixel signal Dp obtained in the pixel array unit 310 is transmitted to the processor 211 via the router 412.
  • congestion in data transmission from the pixel array unit 310 to each processor 211 can be reduced compared to when the pixel array and the multiprocessor are connected via a conventional serializer. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
  • the router 412 is connected to each light receiving pixel P or row readout circuit of the corresponding pixel array 310A and each corresponding processor 211, and is further connected to multiple adjacent routers 412. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
  • the destination of the digital signal obtained by the digital converter is determined by the router 412 based on data about the operating state of the digital converter. This allows data transmission according to the degree of congestion of each processor 211. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using SNN.
  • a control signal ctl1 is generated based on data about the operating state of neurons in the processor 211, and the generated control signal ctl1 is transmitted from the router 412 to the pixel array unit 310.
  • congestion in data transmission from the pixel array unit 310 to each processor 211 can be reduced compared to when the pixel array and multiprocessor are connected via a conventional serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
  • Fig. 13 shows an example of functional blocks of the pixel array units 110, 110A, and 310A.
  • the pixel array units 110, 110A, and 310A may include, for example, a pixel array circuit 101, a vertical scanning circuit 102, and a row readout circuit 103, as shown in Fig. 13.
  • the pixel array circuit 101 has a plurality of light-receiving pixels P arranged two-dimensionally in a matrix, for example, as shown in FIG. 13.
  • a vertical signal line is arranged for each pixel column, and a horizontal signal line is arranged for each pixel row.
  • the vertical scanning circuit 102 selects a number of light receiving pixels P for each row via a number of horizontal signal lines, and outputs the signals generated by each light receiving pixel P for one row to the row readout circuit 103 via a number of vertical signal lines.
  • the row readout circuit 103 has an ADC 104 and a horizontal scanning circuit 105, for example, as shown in FIG. 13.
  • the ADC 104 has a number of ADCs 104a, one for each vertical signal line, and each ADC 104a digitally converts the signals acquired from the light receiving pixels P via the vertical signal line.
  • the horizontal scanning circuit 105 sequentially outputs a number of digital signals (pixel data Dp) obtained from each ADC 104a for each pixel row.
  • the horizontal scanning circuit 105 outputs raster data including a number of pixel data Dp for one row to the routers 120, 120A, and 412.
  • Each router 212, 412 transmits data obtained from routers 120, 120A, 310A, for example, to the corresponding processor 211 using a specified communication standard.
  • Each router 212, 412 transmits, for example, a plurality of pixel data Dp for one frame (all light-receiving pixels P) obtained from routers 120, 120A, 310A, to a plurality of processors 211 together with a frame start (FS) and a frame end (FE).
  • FS frame start
  • FE frame end
  • Each router 212, 412 outputs, for example, FS, a plurality of pixel data Dp for one frame (all light-receiving pixels P), and FE in that order.
  • the LIF unit 211d disables leakage (decrement) over time in the intermediate state during the leaky integrate and fire process from FS to FE. This disablement does not include the decrement that is performed when the data in each row is negatively connected (negative synaptic weight).
  • the LIF unit 211d may disable decrement for the period from FS to FE, for example, as shown in FIG. 14.
  • the period from FS to FE is expressed as a decrement disabled period ⁇ X. This allows each neuron to process one frame of pixel data Dp at the same time.
  • FIG. 15 shows an example of data transmission.
  • Each router 120, 120A transmits, for example, a plurality of pixel data Dp obtained from each light receiving pixel P in the pixel array unit 110, 110A to the corresponding router 212 using a predetermined communication standard.
  • Each router 120, 120A transmits, for example, a plurality of pixel data Dp obtained from each light receiving pixel P in the pixel array unit 110, 110A to the corresponding router 212 together with a frame start (FS) and a frame end (FE).
  • FS frame start
  • FE frame end
  • Each router 120, 120A may, for example, sequentially transmit an FS, a plurality of raster data, and an FE to the router 212 as shown in FIG. 15.
  • the router 212 may obtain the address of the neuron to which the data obtained from the routers 120 and 120A is to be transmitted, for example, based on the TBL. Next, the router 212 may transmit the obtained address to the processor 211, as shown in FIG. 15, by associating it with FS and a plurality of pixel data Dp and FE for one frame (all light receiving pixels P).
  • FIG. 16 shows an example of data transmission.
  • each router 120, 120A may sequentially transmit FS and the raster data of the first row, the raster data of each row from the second row to the (last row - 1) row, and the raster data of the last row and FE to the router 212.
  • the router 212 may obtain the address of the neuron to which the data obtained from the router 120, 120A is to be transmitted, for example, based on the TBL.
  • the router 212 may transmit the obtained address to the processor 211, as shown in FIG. 16, by associating it with FS and each pixel data Dp of one pixel row, each pixel data Dp of the second row to the (last row - 1) row, and FE and each pixel data Dp of the last row.
  • the processor 211 When the processor 211 acquires FS, one frame's worth of pixel data Dp, and FE in the data format shown in FIG. 15 or FIG. 16, for example, it disables decrement during the period from when FS is acquired until when FE is acquired (decrement disabled period ⁇ X). This allows each processor 211 to process one frame's worth of multiple pixel data Dp at the same time.
  • [Variation B] 17 shows a modified example of the functional blocks of the processor unit 200.
  • the processor unit 200 may have, for example, a core array unit 210, a GlobalFS distribution unit 220, and a GlobalFE distribution unit 230, as shown in FIG.
  • the GlobalFS distribution unit 220 transmits an FS to all processors 211.
  • the GlobalFS distribution unit 220 transmits an FS to all processors 211 before the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 starts (i.e., before the processors 211 start reading data).
  • the GlobalFS distribution unit 220 receives a signal (hereinafter referred to as an "input start signal") indicating the start of input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 from the routers 120, 120A
  • the GlobalFS distribution unit 220 transmits an FS to all processors 211.
  • the routers 120, 120A transmit the input start signal to the GlobalFS distribution unit 220, for example, immediately before the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 starts.
  • the GlobalFE distribution unit 230 transmits an FE to all processors 211.
  • the GlobalFE distribution unit 230 transmits an FE to all processors 211 after the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 is completed (i.e., after the data is read by the processors 211 is completed).
  • the GlobalFE distribution unit 230 receives a signal indicating the input completion of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 (hereinafter referred to as the "input completion signal" from the router 120, 120A
  • the GlobalFE distribution unit 230 transmits an FE to all processors 211.
  • the router 120, 120A transmits the input completion signal to the GlobalFE distribution unit 230 at the same time (or immediately after) the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 is completed.
  • Fig. 18 shows a modified example of the functional blocks of the router 120A shown in Fig. 8.
  • Fig. 19 shows a modified example of the functional blocks of the router 412 shown in Fig. 11.
  • the pixel array unit 110, 310 is composed of a plurality of pixel array units 110A, 310A, and pixel data Dp obtained from the pixel array unit 110A, 310A is input to the input port 121, 412a.
  • the pixel data Dp input from the pixel array unit 110A, 310A is input directly to the destination address determination unit 123, 412c without passing through the FIFO memory unit 122, 412b.
  • the destination address determination unit 123, 412c determines whether or not to output the pixel data Dp obtained from the pixel array unit 110A, 310A based on the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f.
  • the destination address determination unit 123, 412c compares the address obtained based on the routing table (TBL) with the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f.
  • the transmission prohibition data 127, 412g is data in which, for example, "1" indicating transmission possible or "0" indicating transmission prohibited is associated with each address, as shown in FIG. 20(A).
  • the transmission prohibition control unit 126, 412f generates transmission prohibition data 127, 412g based on data about the operating state of the neuron in the processor 211 corresponding to the router 212, 412.
  • the transmission prohibition control unit 126, 412f stores the generated transmission prohibition data 127, 412g in a specified memory.
  • the destination address determination unit 123, 412c prohibits the output of pixel data Dp obtained from the pixel array unit 110A, 310A.
  • the destination address determination unit 123, 412c outputs the pixel data Dp obtained from the pixel array unit 110A, 310A to the arbiter 124, 412d.
  • the destination address determination unit 123, 412c permits output to addresses corresponding to "1" in the transmission prohibited data 127, 412g and prohibits output to addresses corresponding to "0" in the transmission prohibited data 127, 412g.
  • the transmission prohibition control unit 126, 412f may reset the transmission prohibition data 127, 412g after a predetermined time has elapsed.
  • the destination address determination unit 123, 412c may redirect the output destination of the pixel data Dp obtained from the pixel array unit 110A, 310A to a neuron having an address corresponding to transmission permitted in the transmission prohibited data 127, 412g.
  • This detouring neuron has a function of auxiliary representation of attention data corresponding to the neuron for which transmission is prohibited.
  • the processor 211 may output the data indicated by the neuron for which transmission is prohibited and the data indicated by the detouring neuron (attention data) in association with each other. In this case, it becomes possible to provide new information processing.
  • Fig. 21 shows a modified example of the functional blocks of the router 120A shown in Fig. 18.
  • Fig. 22 shows a modified example of the functional blocks of the router 412 shown in Fig. 19.
  • the transmission prohibition control units 126, 412f may transmit a control signal ctl3 generated based on the transmission prohibition data 127, 412g to at least one of the vertical scanning circuit 102 and the horizontal scanning circuit 105, for example, as shown in Figs. 21 and 22.
  • the vertical scanning circuit 102 may determine whether or not to select (output data) each line in the pixel array circuit 101 based on the control signal ctl3 input from the transmission prohibition control unit 126, 412f. The vertical scanning circuit 102 may select each line except for the lines for which selection (output data) is prohibited.
  • the horizontal scanning circuit 105 may determine whether or not to output each line of pixel data Dp obtained from the pixel array circuit 101 based on the control signal ctl3 input from the transmission prohibition control units 126 and 412f.
  • the horizontal scanning circuit 105 may output each pixel data Dp except for pixel data Dp for which data output is prohibited.
  • 23(A), 23(B), 23(C), and 23(D) are schematic diagrams showing whether data output is possible based on the control signal ctl3.
  • the pixel array unit 110A, 310A may determine whether data output is possible from each light-receiving pixel P in the pixel array circuit 101 for each line based on the control signal ctl3. At this time, the pixel array unit 110A, 310A outputs pixel data Dp for each line except for lines for which selection (data output) is prohibited, for example, as shown in FIG. 23(A).
  • the pixel array unit 110A, 310A may determine whether data output is possible from the pixel array circuit 101 for each light-receiving pixel P based on the control signal ctl3. At this time, the pixel array units 110A and 310A may output each pixel data Dp except for the pixel data Dp for which data output is prohibited, as shown in, for example, Figures 23(B), 23(C), and 23(D).
  • Fig. 24 shows a modified example of the functional blocks of cell C in Fig. 6.
  • Fig. 25 shows a modified example of the functional blocks of cell C in Fig. 12.
  • the processor 211 may further include a counter 211f, a transmission prohibition control unit 211g, and transmission prohibition data 211h, for example, as shown in Figs. 24 and 25.
  • the counter 211f counts the number of spike signals input from the neuron I/O unit 211a for each neuron destination address.
  • the transmission prohibition control unit 211g writes the neuron destination address corresponding to the count number that exceeds the predetermined threshold into the transmission prohibition data 211h.
  • the transmission prohibition data 211h is data in which either "1" indicating transmission permitted or "0" indicating transmission prohibited is associated with each address.
  • the router 212, 412 determines whether or not to output pixel data Dp obtained from the router 120, 120A or pixel array unit 310A based on the transmission prohibition data 211h.
  • the router 212, 412 compares, for example, an address obtained based on a routing table (TBL) with the transmission prohibition data 211h.
  • TBL routing table
  • the router 212, 412 prohibits the output of pixel data Dp obtained from the router 120, 120A or pixel array unit 310A.
  • the router 212, 412 outputs the pixel data Dp obtained from the router 120, 120A or pixel array unit 310A to the processor 211.
  • the transmission prohibition control unit 211g may reset the transmission prohibition data 211h after a predetermined time has elapsed.
  • the destination address determination unit 212c, 412c may divert the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A to a neuron at an address corresponding to transmission permitted in the transmission prohibited data 211h.
  • This diversion destination neuron has a function of auxiliary representing attention data corresponding to the neuron for which transmission is prohibited.
  • the processor 211 may output the data indicated by the neuron for which transmission is prohibited and the data indicated by the diversion destination neuron (attention data) in association with each other. In this case, it becomes possible to provide new information processing.
  • the processor 211 may change the weight value in the weight storage memory unit 211c based on the time difference value (the difference between the count values for each predetermined period) of the count values stored in the counter 211f.
  • the degree of increase in the count value can be adjusted, and therefore the frequency of detours can be adjusted.
  • congestion in data transmission from the routers 212 and 412 to the processor 211 can be reduced. Therefore, a further improvement in the processing speed can be achieved in signal processing using an SNN.
  • FIG. 26 shows an example of the operation state of a plurality of neurons in the cell C of FIG. 6, FIG. 12, FIG. 24, and FIG. 25.
  • FIG. 26 illustrates the operation state of each neuron in the membrane potential memory unit 211e.
  • some neurons (hereinafter, "main neurons") correspond to the plurality of light receiving pixels P in the pixel array units 110A and 310A.
  • the plurality of neurons other than the main neurons correspond to the detouring destination neurons of the main neurons.
  • FIG. 26 a plurality of neurons whose operation state is transmission prohibited and a plurality of neurons whose operation state is transmission enabled are illustrated.
  • the plurality of neurons whose operation state is transmission prohibited correspond to the main neurons.
  • the plurality of neurons whose operation state is transmission enabled correspond to the detouring destination neurons.
  • the destination address determination unit 212c, 412c may route pixel data Dp obtained from the router 120, 120A or pixel array unit 310A to multiple destination neurons when the operating state of each neuron corresponding to the main neuron is prohibited from transmission.
  • Figures 27(A), 27(B), and 27(C) show an example of a method for bypassing pixel data Dp obtained from router 120, 120A or pixel array unit 310A.
  • the router 212, 412 changes the destination (transfer destination) of the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A to a neuron (a detouring neuron) whose address corresponds to a transmission permitted address.
  • the multiple neurons at the destination correspond to a processing pipeline at the destination.
  • the multiple neurons at the detouring destination correspond to a processing pipeline at the detouring destination.
  • the router 212, 412 outputs transmission data DA to the processor 211, in which the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A corresponds to the destination and detouring addresses.
  • the neuron I/O unit 211a outputs pixel data Dp contained in the transmission data DA received from the routers 212 and 412 as a spike signal to the product-sum calculation unit 211b.
  • the neuron I/O unit 211a outputs the pixel data Dp received from the router 212 to the product-sum calculation unit 211b in association with the address of the detouring destination received from the router 212.
  • the product-sum calculation unit 211b multiplies the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each destination address, and performs a product-sum calculation to add the number of input spikes for each detouring destination address.
  • the product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a.
  • the weight value is stored in the weight storage memory unit 211c.
  • the neuron I/O unit 211a stores the value (result of the product-sum calculation) for each address of the detour destination received from the product-sum calculation unit 211b as a membrane potential in the membrane potential memory unit 211d.
  • the routers 212, 412 generate data on the operating state of the neurons in the processor 211 corresponding to the router 212, 412 based on the data (spike signals and addresses) obtained from the LIF unit 211d in the processor 211.
  • the routers 212, 412 generate (update) transmission prohibition data 127, 412g based on the generated data on the operating state of the neurons in the processor 211.
  • the router 212, 412 determines the status (transmission prohibited or transmission possible) of the address corresponding to the transmission prohibition based on the generated (updated) transmission prohibition data 127, 412g.
  • the router 212, 412 compares the address corresponding to the transmission prohibition with the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f. If the address corresponding to the transmission prohibition corresponds to the transmission prohibition in the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f, the router 212, 412, for example, continues to set the status of the address corresponding to the transmission prohibition to transmission prohibited.
  • the router 212, 412 changes the status of the address corresponding to the transmission prohibition to transmission possible.
  • the router 212, 412 may transmit the processing result of the processing pipeline of the detouring destination to each processing pipeline of the transmission destination, for example, as shown in FIG. 27(A).
  • the router 212, 412 may transmit the processing result of the processing pipeline of the detouring destination to one processing pipeline of the transmission destination, for example, as shown in FIG. 27(B).
  • the router 212, 412 may output data on the processing result at the detouring destination as a control signal ctl, for example, as shown in FIG. 27(C).
  • pixel data Dp obtained from routers 120, 120A or pixel array unit 310A is diverted to multiple neurons at the diverting destination. This makes it possible to reduce congestion in data transmission from pixel array unit 310 to each processor 211. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
  • Fig. 28 shows a modified example of the imaging device 1000 of Fig. 1.
  • the pixel array section 110 may be configured to include a plurality of light receiving pixels P1 arranged two-dimensionally and a plurality of light receiving pixels P2 arranged two-dimensionally, for example, as shown in Fig. 28.
  • the plurality of light receiving pixels P1 and the plurality of light receiving pixels P2 are arranged alternately in the row direction and the column direction, for example.
  • the light receiving pixel P1 may include, for example, a CMOS element or a SPAD element.
  • the first pixel array consisting of a plurality of light receiving pixels P1 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P1, and outputs a plurality of digital signals (pixel data Dp1) to the router 120.
  • the first pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120 at a predetermined period Ta.
  • the router 120 When the router 120 acquires a plurality of pixel data Dp1 from the pixel array unit 110, it transmits transmission data DA including the acquired plurality of pixel data Dp1 to each core C, and outputs the acquired plurality of pixel data Dp1 to the encoder 510 as digital visible light image data Iout1 (for example, RGB image data).
  • the light receiving pixel P2 may include, for example, an EVS element.
  • the second pixel array consisting of a plurality of light receiving pixels P2 detects light in the visible wavelength band incident from the outside with the plurality of light receiving pixels P2, and outputs a plurality of digital signals (pixel data Dp2).
  • the second pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120 at a predetermined period Tb ( ⁇ Ta).
  • Tb predetermined period
  • the router 120 acquires the plurality of pixel data Dp2 from the pixel array section 110, it transmits transmission data DA including the acquired plurality of pixel data Dp2 to each core C, and outputs the acquired plurality of light receiving pixels P2 to the encoder 510 as digital EVS image data Iout2.
  • the pixel array unit 110 outputs visible light image data Iout1 (e.g., RGB image data) to the router 120, and then outputs one or more EVS image data Iout2 to the router 120.
  • the pixel array unit 110 outputs, for example, visible light image data Iout1 (e.g., RGB image data) and one or more EVS image data Iout2 to the router 120 at a period Ta.
  • the encoder 510 encodes the input visible light image data Iout1 (e.g., RGB image data) and outputs the resulting feature image data C1 to the transmitter 520.
  • the encoder 510 encodes the input EVS image data Iout2 and outputs the resulting feature image data C2 to the transmitter 520.
  • the information processing device 3000 includes a receiving unit 3100 capable of communicating with the transmitting unit 520, and a decoder 3200 that decodes the feature amount image data C1 and feature amount image data C2 acquired by the receiving unit 3100.
  • the decoder 3200 generates restored visible light image data Iout1' (e.g., RGB image data) by decoding the feature amount image data C1.
  • the decoder 3200 generates restored EVS image data Iout2' by decoding the feature amount image data C2.
  • the information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3200. For example, as shown in FIG. 29, the data processing unit 3300 generates complementary visible light image data Iout2'' based on the restorable visible light image data Iout1' and the restored EVS image data Iout2'.
  • the complementary visible light image data Iout2'' is data that complements the multiple restored visible light image data Iout1' that are generated periodically.
  • the pixel array section 110 is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout1 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2, to the router 120. As a result, it is possible to reduce the amount of data transmitted from the imaging device 1000 to the information processing device 3000.
  • Fig. 30 shows a modified example of the imaging device 1000 of Fig. 1.
  • the pixel array unit 110 may be provided with an encoder 530 and a communication unit 540 instead of the encoder 510 and the transmission unit 520, as shown in Fig. 30.
  • the encoder 530 encodes the data Dout3 obtained by the core array unit 210 based on the visible light image data Iout1, and outputs the resulting feature image data C3 to the communication unit 540.
  • the encoder 530 encodes the data Dout4 obtained by the core array unit 210 based on the EVS image data Iout2, and outputs the resulting feature image data C4 to the communication unit 540.
  • the information processing device 3000 includes a receiving unit 3400 capable of communicating with the communication unit 540, and a decoder 3500 that decodes the feature amount image data C3 and feature amount image data C4 acquired by the receiving unit 3400.
  • the decoder 3500 generates restored visible light image data Iout3' (e.g., RGB image data) by decoding the feature amount image data C3.
  • the decoder 3500 generates restored EVS image data Iout4' by decoding the feature amount image data C4.
  • the information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3500.
  • the data processing unit 3300 generates complementary visible light image data Iout4'' based on, for example, the restorable visible light image data Iout3' and the restored EVS image data Iout4'.
  • the complementary visible light image data Iout4'' is data that complements multiple restored visible light image data Iout4' that are generated periodically.
  • the pixel array section 110 is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout3 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2, to the router 120. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
  • [Variation I] 31 shows a modified example of the imaging device 1000 of FIG. 1.
  • the pixel array section 110 may further include an encoder 530 and a communication section 540, for example, as shown in FIG. 31.
  • the visible light image data Iout1 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120.
  • the visible light image data Iout3 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120.
  • the amount of data transmission from the imaging device 1000 to the information processing device 3000 can be reduced.
  • Fig. 32 shows a modified example of the imaging device 1000 of Fig. 7.
  • the pixel array section 110A may be configured to include a plurality of light receiving pixels P1 arranged two-dimensionally and a plurality of light receiving pixels P2 arranged two-dimensionally, for example, as shown in Fig. 32.
  • the plurality of light receiving pixels P1 and the plurality of light receiving pixels P2 are arranged alternately in the row direction and the column direction, for example.
  • the light receiving pixel P1 may include, for example, a CMOS element or a SPAD element.
  • the first pixel array consisting of a plurality of light receiving pixels P1 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P1, and outputs a plurality of digital signals (pixel data Dp1) to the router 120A.
  • the first pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120A at a predetermined period Ta.
  • the router 120A When the router 120A acquires a plurality of pixel data Dp1 from the pixel array section 110A, it transmits transmission data DA including the acquired plurality of pixel data Dp1 to the router 210A, and outputs the acquired plurality of pixel data Dp1 to the encoder 510 as digital visible light image data Iout1 (for example, RGB image data).
  • transmission data DA including the acquired plurality of pixel data Dp1 to the router 210A
  • digital visible light image data Iout1 for example, RGB image data
  • the light receiving pixel P2 may include, for example, an EVS element.
  • the second pixel array consisting of a plurality of light receiving pixels P2 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P2 and outputs a plurality of digital signals (pixel data Dp2).
  • the second pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120A at a predetermined period Tb ( ⁇ Ta).
  • the router 120A When the router 120A acquires the plurality of pixel data Dp2 from the pixel array section 110A, it transmits transmission data DA including the acquired plurality of pixel data Dp2 to the router 210A and outputs the acquired plurality of light receiving pixels P2 to the encoder 510 as digital EVS image data Iout2.
  • the pixel array unit 110A outputs visible light image data Iout1 (e.g., RGB image data) to the router 120A, and then outputs one or more EVS image data Iout2 to the router 120A.
  • the pixel array unit 110A outputs, for example, visible light image data Iout1 (e.g., RGB image data) and one or more EVS image data Iout2 to the router 120A at a period Ta.
  • the encoder 510 encodes the input visible light image data Iout1 (e.g., RGB image data) and outputs the resulting feature image data C1 to the transmitter 520.
  • the encoder 510 encodes the input EVS image data Iout2 and outputs the resulting feature image data C2 to the transmitter 520.
  • the information processing device 3000 includes a receiving unit 3100 capable of communicating with the transmitting unit 520, and a decoder 3200 that decodes the feature amount image data C1 and feature amount image data C2 acquired by the receiving unit 3100.
  • the decoder 3200 generates restored visible light image data Iout1' (e.g., RGB image data) by decoding the feature amount image data C1.
  • the decoder 3200 generates restored EVS image data Iout2' by decoding the feature amount image data C2.
  • the information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3200. For example, as shown in FIG. 29, the data processing unit 3300 generates complementary visible light image data Iout2'' based on the restorable visible light image data Iout1' and the restored EVS image data Iout2'.
  • the complementary visible light image data Iout2'' is data that complements the multiple restored visible light image data Iout1' that are generated periodically.
  • the pixel array section 110A is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout1 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2, to the router 120A. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
  • Fig. 33 shows a modified example of the imaging device 1000 of Fig. 1.
  • the pixel array unit 110A may be provided with an encoder 530 and a communication unit 540 instead of the encoder 510 and the transmission unit 520, as shown in Fig. 33.
  • the encoder 530 encodes the data Dout3 obtained by the core array unit 210A based on the visible light image data Iout1, and outputs the resulting feature image data C3 to the communication unit 540.
  • the encoder 530 encodes the data Dout4 obtained by the core array unit 210A based on the EVS image data Iout2, and outputs the resulting feature image data C4 to the communication unit 540.
  • the information processing device 3000 includes a receiving unit 3400 capable of communicating with the communication unit 540, and a decoder 3500 that decodes the feature amount image data C3 and feature amount image data C4 acquired by the receiving unit 3400.
  • the decoder 3500 generates restored visible light image data Iout3' (e.g., RGB image data) by decoding the feature amount image data C3.
  • the decoder 3500 generates restored EVS image data Iout4' by decoding the feature amount image data C4.
  • the information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3500.
  • the data processing unit 3300 generates complementary visible light image data Iout4'' based on, for example, the restorable visible light image data Iout3' and the restored EVS image data Iout4'.
  • the complementary visible light image data Iout4'' is data that complements multiple restored visible light image data Iout4' that are generated periodically.
  • the pixel array section 110A is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout3 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2, to the router 120A. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
  • FIG. 34 shows a modified example of the imaging device 1000 of FIG. 1.
  • the pixel array section 110A may further include an encoder 530 and a communication section 540, for example, as shown in FIG. 34.
  • the visible light image data Iout1 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120A.
  • the visible light image data Iout3 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120A.
  • the amount of data transmission from the imaging device 1000 to the information processing device 3000 can be reduced.
  • imaging device 35 shows an example of use of the imaging device 1 according to the above embodiment and its modified example.
  • the imaging device 1000 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as described below.
  • - Devices for taking images for viewing such as digital cameras and mobile devices with camera functions.
  • - Devices for traffic purposes such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles.
  • - Devices for home appliances such as televisions, refrigerators, and air conditioners that take images of users' gestures and operate the equipment according to those gestures.
  • - Devices for medical and healthcare purposes such as endoscopes and devices that take images of blood vessels by receiving infrared light.
  • - Devices for security purposes such as surveillance cameras for crime prevention and cameras for person authentication.
  • - Devices for beauty purposes such as skin measuring devices that take images of the skin and microscopes that take images of the scalp.
  • - Devices for sports purposes such as action cameras and wearable cameras for sports purposes.
  • - Devices for agriculture such as cameras for monitoring the condition of fields and crops.
  • the technology according to the present disclosure can be applied to various products.
  • the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot.
  • FIG. 36 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology disclosed herein can be applied.
  • the vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001.
  • the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050.
  • Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio/video output unit 12052, and an in-vehicle network I/F (interface) 12053.
  • the drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs.
  • the drive system control unit 12010 functions as a control device for a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force for the vehicle.
  • the body system control unit 12020 controls the operation of various devices installed in the vehicle body according to various programs.
  • the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, tail lamps, brake lamps, turn signals, and fog lamps.
  • radio waves or signals from various switches transmitted from a portable device that replaces a key can be input to the body system control unit 12020.
  • the body system control unit 12020 accepts the input of these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.
  • the outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000.
  • the image capturing unit 12031 is connected to the outside-vehicle information detection unit 12030.
  • the outside-vehicle information detection unit 12030 causes the image capturing unit 12031 to capture images outside the vehicle and receives the captured images.
  • the outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, or characters on the road surface based on the received images.
  • the imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received.
  • the imaging unit 12031 can output the electrical signal as an image, or as distance measurement information.
  • the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared light.
  • the in-vehicle information detection unit 12040 detects information inside the vehicle.
  • a driver state detection unit 12041 that detects the state of the driver is connected.
  • the driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the driver's degree of fatigue or concentration based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.
  • the microcomputer 12051 can calculate the control target values of the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040, and output a control command to the drive system control unit 12010.
  • the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.
  • ADAS Advanced Driver Assistance System
  • the microcomputer 12051 can also control the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040, thereby performing cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on the driver's operation.
  • the microcomputer 12051 can also output control commands to the body system control unit 12020 based on information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching high beams to low beams.
  • the audio/image output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying the occupants of the vehicle or the outside of the vehicle of information.
  • an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices.
  • the display unit 12062 may include, for example, at least one of an on-board display and a head-up display.
  • FIG. 37 shows an example of the installation position of the imaging unit 12031.
  • the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.
  • the imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100.
  • the imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100.
  • the imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100.
  • the imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100.
  • the images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.
  • FIG. 37 shows an example of the imaging ranges of the imaging units 12101 to 12104.
  • Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose
  • imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively
  • imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door.
  • an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.
  • At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information.
  • at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple imaging elements, or an imaging element having pixels for detecting phase differences.
  • the microcomputer 12051 can obtain the distance to each solid object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and can extract as a preceding vehicle, in particular, the closest solid object on the path of the vehicle 12100 that is traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km/h or faster). Furthermore, the microcomputer 12051 can set the inter-vehicle distance that should be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving, which runs autonomously without relying on the driver's operation.
  • automatic braking control including follow-up stop control
  • automatic acceleration control including follow-up start control
  • the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data to automatically avoid obstacles.
  • the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see.
  • the microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of a collision, it can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by forcibly decelerating or steering the vehicle to avoid a collision via the drive system control unit 12010.
  • At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays.
  • the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured image of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian.
  • the audio/image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian.
  • the audio/image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.
  • the technology disclosed herein can be applied to the imaging unit 12031 of the configurations described above.
  • the imaging device mounted on the vehicle can increase the processing speed of captured images.
  • the vehicle control system 12000 can quickly achieve functions such as vehicle collision avoidance or collision mitigation, following driving based on the distance between vehicles, vehicle speed maintenance driving, vehicle collision warning, and vehicle lane departure warning.
  • the present technology can be configured as follows: According to the present technology configured as follows, it is possible to increase the processing speed of captured images. (1) a pixel array in which a plurality of light receiving pixels are arranged two-dimensionally; A multi-processor in which multiple processors configured with SNN (Spiking Neural Network) hardware are arranged in two dimensions; a single-layer or multi-layer router connected to the pixel array and the multiprocessor. (2) The imaging device according to (1), wherein the single-layer or multi-layer router is connected to the plurality of light receiving pixels and the plurality of processors.
  • SNN Spiking Neural Network
  • the light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel, The imaging device according to (2), wherein the single-layer or multi-layer router transmits the digital signals obtained by the digital converter to the processors.
  • the pixel array has a readout circuit that reads out the light receiving pixels row by row, The imaging device according to any one of (1) to (4), wherein the single-level or multilevel router is connected to a readout circuit that reads out the light receiving pixels row by row, and to the processors.
  • the light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel
  • the single-layer or multi-layer router transmits the digital signals obtained from the light-receiving pixels together with a frame start and a frame end to the processors;
  • the multiple hierarchical routers are provided, the plurality of hierarchical routers include a first router in a first hierarchy assigned to the plurality of light receiving pixels, and a plurality of second routers in a second hierarchy assigned to each of the processors,
  • the imaging device according to any one of claims 1 to 5, wherein the first router is connected to each of the second routers.
  • the light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel, The first router transmits data about the operational status of the digital converter to each of the second routers; The imaging device described in (10), wherein the second router determines a destination of the digital signal obtained by the digital converter based on data regarding an operating state of the digital converter obtained from the first router.
  • the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis; The imaging device according to (9), wherein the first router is connected to the readout circuit and each of the second routers.
  • the light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
  • the first router transmits data about the operational status of the digital converter to each of the second routers;
  • the second router transmits data on the operating state of the neuron in the processor corresponding to the second router to the first router;
  • the multiple hierarchical routers include a first hierarchical plurality of first routers each assigned to a pair of the first group and the second group when the light receiving pixels are divided into a plurality of first groups and the processors are divided into a plurality of second groups, and a second hierarchical plurality of second routers each assigned to a pair of the first group and the second group;
  • the imaging device according to any one of claims 1 to 5, wherein the plurality of second routers are assigned to each of the processors.
  • Each of the first routers is connected to each of the light receiving pixels in the corresponding first group and each of the second routers in the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
  • the light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
  • the first router transmits data about the operational status of the digital converter to each of the second routers;
  • the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis;
  • Each of the first routers is connected to the readout circuit of the corresponding first group and to each of the second routers of the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
  • the light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
  • the first router transmits data about the operational status of the digital converter to each of the second routers;
  • the second router transmits data on the operating state of the neuron in the processor corresponding to the second router to the first router;
  • the imaging device according to any one of (15) to (19), wherein the first router determines a destination of the data obtained from the light receiving pixels based on the data from the second router.
  • the single-layer router is provided, the single-layer router includes a plurality of first routers each assigned to each of the first groups when the plurality of light receiving pixels are divided into a plurality of first groups;
  • the imaging device according to any one of claims 1 to 5, wherein the plurality of first routers are assigned to each of the processors.
  • the light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel, The imaging device according to (22), wherein the first router determines a destination of the digital signal obtained by the digital converter based on data on an operating state of the digital converter.
  • the pixel array has a readout circuit that reads out the light receiving pixels row by row, The imaging device according to (21), wherein each of the first routers is connected to the readout circuit of the corresponding first group and to the corresponding processor, and is further connected to a plurality of adjacent first routers.
  • the light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
  • the imaging device according to any one of claims 24 to 30, wherein the first router determines a destination of the digital signal obtained by the digital converter based on an operation state of the digital converter.
  • the first router generates control data based on an operation state of a neuron in the processor corresponding to the first router, and transmits the generated control data to a pixel array;
  • the imaging device according to any one of (1) to (26), wherein the first chip and the second chip are stacked with the first pad electrode and the second pad electrode overlapping each other.
  • the light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel, the first pad electrode is provided on a surface of the first chip opposite to a light receiving surface, and is connected to a wiring through which a digital signal obtained by the digital converter is output;
  • the pixel array has a readout circuit that reads out the light receiving pixels row by row,
  • the light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
  • the first pad electrode is provided on a surface of the first chip opposite to a light receiving surface, and is connected to a wiring through which a digital signal obtained by the digital converter is output;
  • the imaging device according to (27) wherein the second pad electrode is provided on a surface of the second chip and is connected to an input end of the single-layer or multi-layer router.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Solid State Image Pick-Up Elements (AREA)

Abstract

An imaging device according to an aspect of the present invention is provided with: a pixel array unit including a plurality of photoreceptor pixels that are disposed two-dimensionally; a multiprocessor unit including a plurality of processors that are disposed two-dimensionally and are configured of SNN hardware; and a single layer or a plurality of layers of routers that are connected to the pixel array unit and the multiprocessor unit.

Description

撮像装置Imaging device
 本開示は、スパイキングニューラルネットワーク(SNN:Spiking Neural Network)・ハードウェアを利用した撮像装置に関する。 This disclosure relates to an imaging device that utilizes Spiking Neural Network (SNN) hardware.
 近年、ニューラルネットワーク(NN)を用いた信号処理が、ますます機能性と応用を拡げている。その一方で、NNには、処理速度や消費電力に課題がある。この対策として、ニューロモーフィック・ハードウェアを用いることが提案されている。ニューロモーフィック・ハードウェアとして、例えば、スパイキングニューラルネットワーク(SNN)・ハードウェアが挙げられる。 In recent years, signal processing using neural networks (NNs) has been expanding its functionality and applications. However, NNs have issues with processing speed and power consumption. To address these issues, the use of neuromorphic hardware has been proposed. An example of neuromorphic hardware is spiking neural network (SNN) hardware.
 SNNは、従来のNNとは異なり、情報伝達手段としてスパイク信号を用い、個々のニューロンが入力スパイクに応じて変化する中間状態を保持することで、スパイク信号を非同期的に処理可能にしたNNの一形態である。SNNは、このような特徴により、従来のNNと比べて処理速度や消費電力の改善に寄与するとされる。SNNを用いた信号処理については、例えば、特許文献1に開示されている。 Unlike conventional NNs, SNNs use spike signals as a means of transmitting information, and are a form of NN that allows spike signals to be processed asynchronously by maintaining intermediate states in which individual neurons change according to input spikes. Due to these characteristics, SNNs are said to contribute to improved processing speed and power consumption compared to conventional NNs. Signal processing using SNNs is disclosed, for example, in Patent Document 1.
特開2021-13048号公報JP 2021-13048 A
 ところで、SNNを用いた信号処理において、更なる処理速度の向上が望まれている。処理速度を向上することの可能な撮像装置を提供することが望ましい。 Incidentally, there is a demand for further improvements in processing speed in signal processing using SNNs. It would be desirable to provide an imaging device that can improve processing speed.
 本開示の一側面に係る撮像装置は、二次元配置された複数の受光画素を有する画素アレイ部と、二次元配置された、SNN・ハードウェアによって構成された複数のプロセッサを有するマルチプロセッサ部と、画素アレイ部とマルチプロセッサ部とに接続される単一階層もしくは複数階層のルータとを備えた。 An imaging device according to one aspect of the present disclosure includes a pixel array section having a plurality of light receiving pixels arranged two-dimensionally, a multiprocessor section having a plurality of processors arranged two-dimensionally and configured by SNN hardware, and a single-level or multi-level router connected to the pixel array section and the multiprocessor section.
 本開示の一側面に係る撮像装置では、単一階層もしくは複数階層のルータが、複数の受光画素を有する画素アレイ部と、SNN・ハードウェアによって構成された複数のプロセッサを有するマルチプロセッサ部とに接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイからマルチプロセッサへのデータ送信において輻輳が低減される。 In an imaging device according to one aspect of the present disclosure, a single-level or multiple-level router is connected to a pixel array section having multiple light-receiving pixels and a multiprocessor section having multiple processors configured by SNN hardware. This reduces congestion in data transmission from the pixel array to the multiprocessor compared to when the pixel array and the multiprocessor are connected via a conventional serializer.
図1は、本開示の第1の実施の形態に係る撮像装置の概略構成例を表す図である。FIG. 1 is a diagram illustrating an example of a schematic configuration of an imaging device according to a first embodiment of the present disclosure. 図2は、図1に示した撮像装置の積層構成例を表す図である。FIG. 2 is a diagram illustrating an example of a layered configuration of the imaging device illustrated in FIG. 図3は、図1に示した受光画素の機能ブロック例を表す図である。FIG. 3 is a diagram illustrating an example of a functional block of the light receiving pixel illustrated in FIG. 図4は、図1に示したセンサ部側のルータの機能ブロック例を表す図である。FIG. 4 is a diagram illustrating an example of a functional block of the router on the sensor unit side illustrated in FIG. 図5は、図1に示したプロセッサ部側のルータの機能ブロック例を表す図である。FIG. 5 is a diagram illustrating an example of functional blocks of a router on the processor unit side illustrated in FIG. 図6は、図1に示したプロセッサの機能ブロック例を表す図である。FIG. 6 is a diagram illustrating an example of functional blocks of the processor illustrated in FIG. 図7は、図2に示した撮像装置の積層構造の一変形例を表すブロック図である。FIG. 7 is a block diagram showing a modified example of the stacked structure of the imaging device shown in FIG. 図8は、図7に示した受光画素側のルータの機能ブロック例を表す図である。FIG. 8 is a diagram illustrating an example of functional blocks of the router on the light receiving pixel side illustrated in FIG. 図9は、本開示の第2の実施の形態に係る撮像装置の概略構成例を表す図である。FIG. 9 is a diagram illustrating an example of a schematic configuration of an imaging device according to the second embodiment of the present disclosure. 図10は、図9に示した撮像装置の積層構成例を表す図である。FIG. 10 is a diagram illustrating an example of a layered configuration of the imaging device illustrated in FIG. 図11は、図9に示したルータの機能ブロック例を表す図である。FIG. 11 is a diagram illustrating an example of functional blocks of the router illustrated in FIG. 図12は、図9に示したプロセッサの機能ブロック例を表す図である。FIG. 12 is a diagram illustrating an example of functional blocks of the processor illustrated in FIG. 図13は、図7,図10に示した画素アレイ部の機能ブロック例を表す図である。FIG. 13 is a diagram illustrating an example of functional blocks of the pixel array unit illustrated in FIGS. 図14は、図6,図12に示した膜電位メモリ部内の膜電位の時系列の値の一例を表す図である。FIG. 14 is a diagram showing an example of time series values of the membrane potential in the membrane potential memory unit shown in FIGS. 図15は、送信データの一例を表す図である。FIG. 15 is a diagram illustrating an example of transmission data. 図16は、送信データの一例を表す図である。FIG. 16 is a diagram illustrating an example of transmission data. 図17は、図9に示したプロセッサ部の機能ブロック例を表す図である。FIG. 17 is a diagram illustrating an example of functional blocks of the processor unit illustrated in FIG. 図18は、図8に示したルータの機能ブロックの一変形例を表す図である。FIG. 18 is a diagram showing a modified example of the functional blocks of the router shown in FIG. 図19は、図11に示したルータの機能ブロックの一変形例を表す図である。FIG. 19 is a diagram showing a modified example of the functional blocks of the router shown in FIG. 図20は、図19に示した伝送禁止テーブルの一例を表す図である。FIG. 20 is a diagram showing an example of the transmission prohibition table shown in FIG. 図21は、図8に示したルータの機能ブロックの一変形例を表す図である。FIG. 21 is a diagram showing a modified example of the functional blocks of the router shown in FIG. 図22は、図8に示したルータの機能ブロックの一変形例を表す図である。FIG. 22 is a diagram showing a modified example of the functional blocks of the router shown in FIG. 図23(A),図23(B),図23(C),図23(D)は、制御信号ctl3に基づいて決定されたデータ出力の可否を模式的に表したものである。23A, 23B, 23C, and 23D are schematic diagrams showing whether data output is possible or not, determined based on the control signal ctl3. 図24は、図6のセルの機能ブロックの一変形例を表す図である。FIG. 24 is a diagram showing a modification of the functional blocks of the cell of FIG. 図25は、図12のセルの機能ブロックの一変形例を表す図である。FIG. 25 is a diagram showing a modified example of the functional blocks of the cell of FIG. 図26は、プロセッサ内の複数のニューロンの動作状態の一例を表す図である。FIG. 26 is a diagram showing an example of the operating state of a plurality of neurons in a processor. 図27は、プロセッサ内におけるデータの迂回の一例を表す図である。FIG. 27 is a diagram illustrating an example of data bypass within a processor. 図28は、図1に示した撮像装置の概略構成の一変形例を表す図である。FIG. 28 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. 図29は、図28に示した撮像装置のルータからの出力画像およびその出力画像のデコード後の画像の一例を表す図である。29A and 29B are diagrams illustrating an example of an output image from the router of the imaging device illustrated in FIG. 28 and an image obtained after the output image is decoded. 図30は、図1に示した撮像装置の概略構成の一変形例を表す図である。FIG. 30 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. 図31は、図1に示した撮像装置の概略構成の一変形例を表す図である。FIG. 31 is a diagram illustrating a modification of the schematic configuration of the imaging device shown in FIG. 図32は、図7に示した撮像装置の概略構成の一変形例を表す図である。FIG. 32 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. 図33は、図7に示した撮像装置の概略構成の一変形例を表す図である。FIG. 33 is a diagram illustrating a modification of the schematic configuration of the imaging device shown in FIG. 図34は、図7に示した撮像装置の概略構成の一変形例を表す図である。FIG. 34 is a diagram showing a modification of the schematic configuration of the imaging device shown in FIG. 撮像装置の使用例を表す説明図である。FIG. 1 is an explanatory diagram illustrating an example of use of an imaging device. 車両制御システムの概略的な構成の一例を示すブロック図である。1 is a block diagram showing an example of a schematic configuration of a vehicle control system; 車外情報検出部及び撮像部の設置位置の一例を示す説明図である。4 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit; FIG.
 以下、本開示を実施するための形態について、図面を参照して詳細に説明する。以下の説明は本開示の一具体例であって、本開示は以下の態様に限定されるものではない。なお、説明は、以下の順序で行う。
1.第1の実施の形態(図1~図6)
2.第1の実施の形態の変形例(図7,図8)
3.第2の実施の形態(図9~図12)
4.各実施の形態に共通の変形例(図13~図34)
5.撮像装置の使用例(図35)
6.移動体への応用例(図36,図37)
Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. The following description is a specific example of the present disclosure, and the present disclosure is not limited to the following embodiment. The description will be given in the following order.
1. First embodiment (FIGS. 1 to 6)
2. Modifications of the First Embodiment (FIGS. 7 and 8)
3. Second embodiment (FIGS. 9 to 12)
4. Modifications common to each embodiment (FIGS. 13 to 34)
5. Examples of use of imaging devices (Figure 35)
6. Examples of applications to moving objects (Figures 36 and 37)
<1.第1の実施の形態>
[構成例]
 図1は、本開示の一実施の形態に係る撮像装置1000の概略構成例を表すものである。撮像装置1000は、例えば、図1に示したように、センサ部100およびプロセッサ部200を備える。センサ部100は、画素アレイ部110およびルータ120を有する。プロセッサ部200は、コアアレイ部210を有する。コアアレイ部210は、二次元配置された複数のコアCを有する。各コアCは、プロセッサ211およびルータ212を有する。コアアレイ部210は、二次元配置された複数のプロセッサ211と、二次元配置された複数のルータ212とを有する。複数のルータ212は、プロセッサ211ごとに1つずつ割り当てられる。二次元配置された複数のプロセッサ211が、本開示の一実施の形態に係る「マルチプロセッサ」の一具体例に相当する。プロセッサ211は、SNN(Spiking Neural Network)・ハードウェアによって構成される。画素アレイ部110と各プロセッサ211とは、ルータ120,212を介して直接、接続される。
1. First embodiment
[Configuration example]
FIG. 1 shows a schematic configuration example of an imaging device 1000 according to an embodiment of the present disclosure. For example, as shown in FIG. 1, the imaging device 1000 includes a sensor unit 100 and a processor unit 200. The sensor unit 100 includes a pixel array unit 110 and a router 120. The processor unit 200 includes a core array unit 210. The core array unit 210 includes a plurality of cores C arranged two-dimensionally. Each core C includes a processor 211 and a router 212. The core array unit 210 includes a plurality of processors 211 arranged two-dimensionally and a plurality of routers 212 arranged two-dimensionally. The plurality of routers 212 are assigned to each processor 211. The two-dimensionally arranged processors 211 correspond to a specific example of a "multiprocessor" according to an embodiment of the present disclosure. The processor 211 is configured by SNN (Spiking Neural Network) hardware. The pixel array unit 110 and each processor 211 are directly connected via routers 120 and 212 .
 画素アレイ部110は、二次元配置された複数の受光画素Pを有する。受光画素Pは、例えば、CMOS(Complementary Metal Oxide Semiconductor)素子、EVS(Event-Based Vision Sensor)素子、または、SPAD(Single Photon Avalanche Diode)素子を含む。受光画素Pは、外部から入射した光を検出することにより検出信号を生成する。画素アレイ部110は、例えば、生成した検出信号を直接、ルータ120に送信してもよい。画素アレイ部110は、例えば、上記検出信号を画素アレイ部110内のADC(Analog to Digital Convertor)もしくはカウンタによってデジタル化し、それにより得られたデジタル信号をルータ120に送信してもよい。画素アレイ部110は、受光画素Pでの光検出に基づいて生成されたデータ(例えば、上記検出信号、または、上記デジタル信号)を画素データDpとしてルータ120に送信する。上記検出信号、または、上記デジタル信号は、いわゆるスパイク信号に相当する。 The pixel array unit 110 has a plurality of light receiving pixels P arranged two-dimensionally. The light receiving pixels P include, for example, a CMOS (Complementary Metal Oxide Semiconductor) element, an EVS (Event-Based Vision Sensor) element, or a SPAD (Single Photon Avalanche Diode) element. The light receiving pixels P generate detection signals by detecting light incident from the outside. The pixel array unit 110 may, for example, directly transmit the generated detection signals to the router 120. The pixel array unit 110 may, for example, digitize the detection signals using an ADC (Analog to Digital Converter) or a counter in the pixel array unit 110, and transmit the resulting digital signals to the router 120. The pixel array unit 110 transmits data (for example, the detection signals or the digital signals) generated based on the light detection at the light receiving pixels P to the router 120 as pixel data Dp. The detection signal or the digital signal corresponds to a so-called spike signal.
 画素アレイ部110は、ルータ120からの制御データctl2に基づいて、各受光画素Pから得られた画素データDpをルータ120に送信してもよい。この制御データctl2には、例えば、画素アレイ部110からのデータ出力の要否についての受光画素Pごとのデータが含まれる。画素アレイ部110は、制御データctl2に基づいて、各受光画素Pから得られた画素データDpの出力の要否を決定し、出力要と決定された受光画素Pの画素データDpをルータ120に送信してもよい。 The pixel array section 110 may transmit pixel data Dp obtained from each light receiving pixel P to the router 120 based on control data ctl2 from the router 120. This control data ctl2 includes, for example, data for each light receiving pixel P regarding whether data output from the pixel array section 110 is required. The pixel array section 110 may determine whether or not it is required to output pixel data Dp obtained from each light receiving pixel P based on the control data ctl2, and transmit the pixel data Dp of the light receiving pixel P determined to require output to the router 120.
 ルータ120は、画素アレイ部110から画素データDpを取得すると、取得した画素データDpに対応する画素アドレスを参照して、画素データDpの送信先のニューロンのアドレスを取得する。ルータ120には、ルーティングテーブルが設けられており、ルータ120は、ルーティングテーブルに基づいて、画素データDpの送信先のニューロンのアドレスを取得する。ルータ120は、必要に応じて、画素データDpを取得したときの時間データ(タイムスタンプ)を生成してもよい。 When the router 120 acquires pixel data Dp from the pixel array unit 110, it references the pixel address corresponding to the acquired pixel data Dp to acquire the address of the neuron to which the pixel data Dp is to be sent. The router 120 is provided with a routing table, and the router 120 acquires the address of the neuron to which the pixel data Dp is to be sent based on the routing table. The router 120 may generate time data (timestamp) when the pixel data Dp was acquired, if necessary.
 ルータ120は、取得したアドレスと画素データDpとを含む送信データDAを各コアCへ送信する。ルータ120は、コアアレイ部210からの制御データctl1に基づいて、送信データDAの送信先を決定してもよい。この制御データctl1には、例えば、コアアレイ部210内の各ニューロンの動作状態についてのデータが含まれる。ニューロンの状態には、例えば、処理待ち(ビジー)か否かについてのデータが含まれる。 The router 120 transmits transmission data DA, which includes the acquired address and pixel data Dp, to each core C. The router 120 may determine the destination of the transmission data DA based on control data ctl1 from the core array unit 210. This control data ctl1 includes, for example, data on the operating state of each neuron in the core array unit 210. The state of a neuron includes, for example, data on whether it is waiting for processing (busy) or not.
 ルータ120は、画素アレイ部110に含まれるデジタル変換器(例えば、後述の読出部113,116,119に含まれるデジタル変換器)の動作状態についてのデータをコアアレイ部210内の各ルータ212に送信してもよい。この動作状態には、例えば、デジタル変換器の動作の有無や、デジタル変換のビット幅、デジタル変換器の動作タイミングもしくは動作周波数が含まれる。 The router 120 may transmit data about the operating state of the digital converters included in the pixel array unit 110 (for example, the digital converters included in the readout units 113, 116, and 119 described below) to each router 212 in the core array unit 210. This operating state may include, for example, whether the digital converter is operating, the bit width of the digital conversion, and the operating timing or operating frequency of the digital converter.
 次に、コアC内のルータ212について説明する。コアアレイ部210内には、複数のルータ212が二次元配置される。ルータ212は、ルータ120から取得した送信データDAに含まれるアドレスを参照して、送信データDAの送信先を決定する。ルータ212には、ルーティングテーブルが設けられており、ルータ212は、ルーティングテーブルに基づいて、送信データDAの送信先を決定する。ルータ212は、ルータ120から取得した送信データDAに含まれるアドレスと、ルータ120から取得した動作状態(画素アレイ部110に含まれるデジタル変換器の動作状態)についてのデータとに基づいて、送信データDAの送信先を決定してもよい。 Next, the router 212 in the core C will be described. A plurality of routers 212 are arranged two-dimensionally in the core array unit 210. The router 212 determines the destination of the transmission data DA by referring to the address included in the transmission data DA acquired from the router 120. The router 212 is provided with a routing table, and determines the destination of the transmission data DA based on the routing table. The router 212 may determine the destination of the transmission data DA based on the address included in the transmission data DA acquired from the router 120 and data on the operating state (the operating state of the digital converter included in the pixel array unit 110) acquired from the router 120.
 ルータ212は、決定した送信先に送信データDAを送信する。決定した送信先が当該ルータ212に対応するプロセッサ211内のニューロンである場合、ルータ212は、送信データDAを当該ルータ212に対応するプロセッサ211に送信する。決定した送信先が当該ルータ212に隣接するルータ212に対応するプロセッサ211内のニューロンである場合、ルータ212は、送信データDAを、当該ルータ212に隣接するルータ212に対応するプロセッサ211に送信する。プロセッサ211は、当該プロセッサ211に対応するルータ212から取得した送信データDAに含まれる画素データに対して、SNNを用いた信号処理を行う。コアアレイ部210は、プロセッサ211における信号処理により得られたデータDoutを外部に出力する。 The router 212 transmits the transmission data DA to the determined destination. If the determined destination is a neuron in the processor 211 corresponding to the router 212, the router 212 transmits the transmission data DA to the processor 211 corresponding to the router 212. If the determined destination is a neuron in the processor 211 corresponding to the router 212 adjacent to the router 212, the router 212 transmits the transmission data DA to the processor 211 corresponding to the router 212 adjacent to the router 212. The processor 211 performs signal processing using SNN on the pixel data included in the transmission data DA acquired from the router 212 corresponding to the processor 211. The core array unit 210 outputs data Dout obtained by signal processing in the processor 211 to the outside.
 ルータ212は、当該ルータ212に対応するプロセッサ211内のLIF部211d(後述)から得られたデータ(スパイク信号およびアドレス)に基づいて、当該ルータ212に対応するプロセッサ211内のニューロンの動作状態についてのデータを生成する。ルータ212は、生成したデータ(当該ルータ212に対応するプロセッサ211内のニューロンの動作状態についてのデータ)をルータ120に送信してもよい。 The router 212 generates data about the operating state of the neuron in the processor 211 corresponding to the router 212 based on data (spike signals and addresses) obtained from the LIF unit 211d (described below) in the processor 211 corresponding to the router 212. The router 212 may transmit the generated data (data about the operating state of the neuron in the processor 211 corresponding to the router 212) to the router 120.
 図2は、撮像装置1000の積層構造の一例を表すものである。画素アレイ部110は、例えば、図2に示したように、センサチップ1000Aによって構成される。センサチップ1000Aにおいて、画素アレイ部110は半導体基板上に形成される。ルータ120およびプロセッサ部200からなる回路が、例えば、図2に示したように、SNNチップ1000Bによって構成される。SNNチップ1000Bにおいて、ルータ120およびプロセッサ部200は半導体基板上に形成される。 FIG. 2 shows an example of a layered structure of the imaging device 1000. The pixel array section 110 is formed, for example, by a sensor chip 1000A as shown in FIG. 2. In the sensor chip 1000A, the pixel array section 110 is formed on a semiconductor substrate. A circuit consisting of the router 120 and the processor section 200 is formed, for example, by an SNN chip 1000B as shown in FIG. 2. In the SNN chip 1000B, the router 120 and the processor section 200 are formed on a semiconductor substrate.
 センサチップ1000Aには、受光画素Pごとに1または複数のパッド電極PE1が設けられる。各パッド電極PE1は、センサチップ1000Aにおいて、受光面とは反対側の表面に設けられ、送信データDAが送信される配線に接続される。SNNチップ1000Bには、受光画素Pごとに1または複数のパッド電極PE2が設けられる。各パッド電極PE2は、SNNチップ1000Bの表面に設けられ、ルータ120の入力端に接続される。センサチップ1000AとSNNチップ1000Bとは、パッド電極PE1とパッド電極PE2とを互いに重ね合わせて積層される。 The sensor chip 1000A is provided with one or more pad electrodes PE1 for each light receiving pixel P. Each pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to wiring through which transmission data DA is transmitted. The SNN chip 1000B is provided with one or more pad electrodes PE2 for each light receiving pixel P. Each pad electrode PE2 is provided on the surface of the SNN chip 1000B, and is connected to the input terminal of the router 120. The sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
 ルータ120の入力端は各受光画素Pに接続され、ルータ120の出力端は各コアC(ルータ212)に接続される。SNNチップ1000Bにおいて、ルータ120は、複数のルータ212が設けられた箇所と対向する箇所に設けられる。画素アレイ部110と各プロセッサ211との間に設けられたルータは、第1階層のルータ120と第2階層の複数のルータ212とにより構成された複数階層(2階層)となっている。 The input terminal of the router 120 is connected to each light receiving pixel P, and the output terminal of the router 120 is connected to each core C (router 212). In the SNN chip 1000B, the router 120 is provided at a location opposite to the location where multiple routers 212 are provided. The router provided between the pixel array unit 110 and each processor 211 is a multi-layer (two-layer) structure consisting of a first-layer router 120 and multiple second-layer routers 212.
 図3(A)~図3(C)は、受光画素Pの構成例を表すものである。 Figures 3(A) to 3(C) show examples of the configuration of a light-receiving pixel P.
 受光画素Pは、CMOS素子を含んでもよい。この場合、受光画素Pは、例えば、図3(A)に示したように、光電変換部111および電荷蓄積部112を含む。光電変換部111は、例えば、フォトダイオードを含んで構成され、センサチップ1000Aの受光面に入射した光に対して光電変換を行って受光量に応じた電荷を発生する。電荷蓄積部112は、例えば、フォトダイオードと電気的に接続された転送トランジスタと、転送トランジスタを介してフォトダイオードから出力された電荷を一時的に保持するフローティングディフュージョンとを含む。電荷蓄積部112は、例えば、電荷蓄積部112に保持された電荷のレベルに応じた電圧の信号を検出信号として出力する。 The light receiving pixel P may include a CMOS element. In this case, the light receiving pixel P includes, for example, a photoelectric conversion unit 111 and a charge storage unit 112 as shown in FIG. 3A. The photoelectric conversion unit 111 includes, for example, a photodiode, and performs photoelectric conversion on light incident on the light receiving surface of the sensor chip 1000A to generate a charge according to the amount of light received. The charge storage unit 112 includes, for example, a transfer transistor electrically connected to the photodiode, and a floating diffusion that temporarily holds the charge output from the photodiode via the transfer transistor. The charge storage unit 112 outputs, for example, a voltage signal according to the level of the charge held in the charge storage unit 112 as a detection signal.
 受光画素Pは、さらに、例えば、図3(A)に示したように、読出部113を有してもよい。読出部113は、電荷蓄積部112に保持された電荷のレベルに応じた電圧の信号(検出信号)をデジタル変換するデジタル変換器と、デジタル変換により得られたデジタル信号(画素データDp)を出力する出力回路とを有する。画素アレイ部110は、例えば、さらに、一行分の複数の画素データDpを含むラスターデータを画素行ごとに複数の受光画素Pから読み出し、読み出したラスターデータをルータ120に送信する行読み出し回路を有してもよい。行読み出し回路は、複数の受光画素Pから得られた複数のデジタル信号(画素データDp)を画素行ごとに順次出力する。複数の受光画素Pから得られた複数のデジタル信号によって、可視光画像データ(例えばRGB画像データ)が生成される。 The light receiving pixel P may further include a readout unit 113, for example, as shown in FIG. 3A. The readout unit 113 includes a digital converter that digitally converts a voltage signal (detection signal) corresponding to the level of the charge stored in the charge storage unit 112, and an output circuit that outputs a digital signal (pixel data Dp) obtained by digital conversion. The pixel array unit 110 may further include a row readout circuit that reads raster data including one row of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the read raster data to the router 120. The row readout circuit sequentially outputs the digital signals (pixel data Dp) obtained from the light receiving pixels P for each pixel row. Visible light image data (for example, RGB image data) is generated by the digital signals obtained from the light receiving pixels P.
 受光画素Pは、EVS素子を含んでもよい。この場合、受光画素Pは、例えば、図3(B)に示したように、光電変換部114および減算部115を含む。光電変換部114は、例えば、フォトダイオードを含んで構成され、センサチップ1000Aの受光面に入射した光に対して光電変換を行って受光量に応じた電荷を発生する。減算部115は、例えば、バッファと、サンプルホールド回路とを含んで構成される。バッファは、フォトダイオードから出力された電荷のレベルに応じた電圧の信号を保持する。サンプルホールド回路は、バッファから供給された信号をサンプリングし、サンプリングされた信号を保持し、その後に、バッファから供給された信号と、サンプルホールド回路に保持された信号との差分に応じた信号を検出信号として出力する。減算部115は、受光画素P内のメモリのように機能する。 The light receiving pixel P may include an EVS element. In this case, the light receiving pixel P includes, for example, a photoelectric conversion unit 114 and a subtraction unit 115, as shown in FIG. 3B. The photoelectric conversion unit 114 includes, for example, a photodiode, and performs photoelectric conversion on the light incident on the light receiving surface of the sensor chip 1000A to generate a charge according to the amount of light received. The subtraction unit 115 includes, for example, a buffer and a sample and hold circuit. The buffer holds a voltage signal according to the level of the charge output from the photodiode. The sample and hold circuit samples the signal supplied from the buffer, holds the sampled signal, and then outputs a signal according to the difference between the signal supplied from the buffer and the signal held in the sample and hold circuit as a detection signal. The subtraction unit 115 functions like a memory in the light receiving pixel P.
 受光画素Pは、さらに、例えば、図3(B)に示したように、読出部116を有してもよい。読出部116は、減算部115から出力された信号(検出信号)をデジタル変換するデジタル変換器と、デジタル変換により得られたデジタル信号(画素データDp)を出力する出力回路とを有する。画素アレイ部110は、例えば、さらに、一行分の複数の画素データDpを含むラスターデータを画素行ごとに複数の受光画素Pから読み出し、読み出したルータ120に送信する行読み出し回路を有してもよい。行読み出し回路は、複数の受光画素Pから得られた複数のデジタル信号(画素データDp)を画素行ごとに順次出力する。複数の受光画素Pから得られた複数のデジタル信号によって、EVS画像データが生成される。 The light receiving pixel P may further include a readout unit 116, for example, as shown in FIG. 3B. The readout unit 116 includes a digital converter that digitally converts the signal (detection signal) output from the subtraction unit 115, and an output circuit that outputs a digital signal (pixel data Dp) obtained by digital conversion. The pixel array unit 110 may further include a row readout circuit that reads out raster data including one row's worth of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the read data to the router 120. The row readout circuit sequentially outputs the digital signals (pixel data Dp) obtained from the light receiving pixels P for each pixel row. EVS image data is generated by the digital signals obtained from the light receiving pixels P.
 受光画素Pは、SPAD素子を含んでもよい。この場合、受光画素Pは、例えば、図3(C)に示したように、SPAD部117およびパルス検出部118を含む。SPAD部117は、例えば、SPAD素子を含んで構成され、ガイガーモードで動作し、SPAD素子のアノードとカソードとの間に降伏電圧(ブレークダウン電圧)以上の負バイアス電圧が印加されている状態でフォトンが入射すると、アバランシェ電流を発生する。パルス検出部118は、例えば、SPAD部117に直列に接続されたクエンチ抵抗と、SPAD部117とクエンチ抵抗との接続ノードに接続されたインバータとを含んで構成される。インバータは、接続ノードの電圧が所定の閾値電圧よりも低いとき(つまりローレベルのとき)、ハイレベルの信号を出力する。インバータは、接続ノードの電圧が所定の閾値電圧以上のとき(つまりハイレベルのとき)、ローレベルの信号を出力する。このように、検出部118は、検出信号としてデジタル信号を出力するデジタル変換器として機能する。 The light receiving pixel P may include a SPAD element. In this case, the light receiving pixel P includes, for example, a SPAD section 117 and a pulse detection section 118, as shown in FIG. 3C. The SPAD section 117 includes, for example, a SPAD element, operates in Geiger mode, and generates an avalanche current when a photon is incident in a state where a negative bias voltage equal to or greater than the breakdown voltage is applied between the anode and cathode of the SPAD element. The pulse detection section 118 includes, for example, a quench resistor connected in series to the SPAD section 117, and an inverter connected to a connection node between the SPAD section 117 and the quench resistor. The inverter outputs a high-level signal when the voltage of the connection node is lower than a predetermined threshold voltage (i.e., when it is at a low level). The inverter outputs a low-level signal when the voltage of the connection node is equal to or greater than a predetermined threshold voltage (i.e., when it is at a high level). In this way, the detection section 118 functions as a digital converter that outputs a digital signal as a detection signal.
 受光画素Pは、さらに、例えば、図3(C)に示したように、読出部119を有してもよい。読出部119は、パルス検出部118から出力された信号をカウントし、カウント値に基づいた信号(画素データDp)を出力するカウンタを有する。画素アレイ部110は、例えば、さらに、一行分の複数の画素データDpを含むラスターデータを画素行ごとに複数の受光画素Pから読み出し、読み出したラスターデータをルータ120に送信する行読み出し回路を有してもよい。行読み出し回路は、複数の受光画素Pから得られた複数のデジタル信号を画素行ごとに順次出力する。複数の受光画素Pから得られた複数のデジタル信号によって、可視光画像データ(例えばRGB画像データ)が生成される。 The light receiving pixel P may further include a readout unit 119, for example, as shown in FIG. 3C. The readout unit 119 includes a counter that counts the signal output from the pulse detection unit 118 and outputs a signal (pixel data Dp) based on the count value. The pixel array unit 110 may further include a row readout circuit that reads out raster data including one row's worth of pixel data Dp from the light receiving pixels P for each pixel row, and transmits the readout raster data to the router 120. The row readout circuit sequentially outputs multiple digital signals obtained from the multiple light receiving pixels P for each pixel row. Visible light image data (e.g., RGB image data) is generated by the multiple digital signals obtained from the multiple light receiving pixels P.
 読出部113,116,119または行読み出し回路には、ルータ120から制御データ(制御データctl2)が入力されてもよい。この場合、読出部113,116,119または行読み出し回路は、ルータ120からの制御データ(制御データctl2)に基づいて、画素データDpをルータ120に出力してもよい。 The readout units 113, 116, and 119 or the row readout circuit may receive control data (control data ctl2) from the router 120. In this case, the readout units 113, 116, and 119 or the row readout circuit may output pixel data Dp to the router 120 based on the control data (control data ctl2) from the router 120.
 ところで、画素アレイ部110が行読み出し回路を有するとする。このとき、センサチップ1000Aにおいて、1または複数のパッド電極PE1が行読み出し回路に接続されてもよい。このとき、ルータ120の入力端は行読み出し回路に接続され、ルータ120の出力端は各コアCに接続される。なお、受光画素Pに設けられるデジタル変換器が、行読み出し回路内に設けられてもよい。 Now, suppose that the pixel array section 110 has a row readout circuit. In this case, in the sensor chip 1000A, one or more pad electrodes PE1 may be connected to the row readout circuit. In this case, the input terminal of the router 120 is connected to the row readout circuit, and the output terminal of the router 120 is connected to each core C. Note that the digital converter provided in the light-receiving pixel P may be provided within the row readout circuit.
(ルータ120)
 図4は、ルータ120の概略構成例を表すものである。ルータ120は、例えば、図4に示したように、inputポート121、FIFO(first-in first-out)メモリ部122、行先アドレス決定部123、アービタ124およびoutputポート125を有する。
(Router 120)
Fig. 4 shows an example of a schematic configuration of the router 120. The router 120 has, for example, an input port 121, a FIFO (first-in first-out) memory unit 122, a destination address determination unit 123, an arbiter 124, and an output port 125, as shown in Fig. 4.
 inputポート121は、各パッド電極PE2に電気的に接続され、画素アレイ部110から送信された複数の画素信号DpをFIFOメモリ部122に出力する。FIFOメモリ部122は、inputポート121から入力された複数の画素信号Dpを一時保存する。FIFOメモリ部122は、FIFOメモリ部122に保存された複数の画素信号Dpを、アービタ124の制御に応じて順次出力する。行先アドレス決定部123は、ルーティングテーブル(TBL)を有し、TBLに基づいて、各画素データDpの送信先のニューロンのアドレスを取得する。行先アドレス決定部123は、画素信号Dpと、取得したアドレスとを対応付けてアービタ124に出力する。 The input port 121 is electrically connected to each pad electrode PE2, and outputs a plurality of pixel signals Dp transmitted from the pixel array section 110 to the FIFO memory section 122. The FIFO memory section 122 temporarily stores the plurality of pixel signals Dp input from the input port 121. The FIFO memory section 122 sequentially outputs the plurality of pixel signals Dp stored in the FIFO memory section 122 under the control of the arbiter 124. The destination address determination section 123 has a routing table (TBL), and acquires the address of the neuron to which each pixel data Dp is to be transmitted based on the TBL. The destination address determination section 123 associates the pixel signal Dp with the acquired address and outputs it to the arbiter 124.
 アービタ124は、複数の受光画素Pのそれぞれから供給される画素信号Dpの出力を要求するリクエストを調停し、その調停結果(即ち、画素信号Dpの出力の許可/不許可)に基づく応答をoutputポート125に出力する。アービタ124は、例えば、制御データctl1に従って調停を行ってもよい。アービタ124は、例えば、制御データctl1に基づいて、送信データDAの送信先を決定してもよい。制御データctl1には、例えば、コアアレイ部210内の各ニューロンの動作状態についてのデータが含まれる。 The arbiter 124 arbitrates requests for output of pixel signals Dp supplied from each of the multiple light receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 125. The arbiter 124 may perform arbitration according to control data ctl1, for example. The arbiter 124 may determine the destination of the transmission data DA based on the control data ctl1, for example. The control data ctl1 includes data on the operating state of each neuron in the core array unit 210, for example.
 アービタ124は、例えば、調停結果に基づいて生成した制御データctl2を画素アレイ部110に出力してもよい。アービタ124は、例えば、制御データctl1に基づいて制御データctl2を生成し、生成した制御データctl2を画素アレイ部110に出力してもよい。制御データctl2には、例えば、画素アレイ部110からのデータ出力の要否についての受光画素Pごとのデータが含まれる。outputポート125は、送信データDAを各コアCへ送信する。 The arbiter 124 may, for example, output the control data ctl2 generated based on the arbitration result to the pixel array unit 110. The arbiter 124 may, for example, generate the control data ctl2 based on the control data ctl1 and output the generated control data ctl2 to the pixel array unit 110. The control data ctl2 includes, for example, data for each light receiving pixel P regarding whether or not data output from the pixel array unit 110 is required. The output port 125 transmits the transmission data DA to each core C.
 ルータ120の構成は、図4に示した構成に限定されるものではない。リクエストの調停を行うアービタや、FIFOメモリを用いたバッファリング、TBLを用いた出力先選定などの機能を備えたものであれば、ルータ120の構成は、図4に示した構成に限定されるものではない。 The configuration of the router 120 is not limited to the configuration shown in FIG. 4. The configuration of the router 120 is not limited to the configuration shown in FIG. 4 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
 図5は、ルータ212の概略構成例を表すものである。ルータ212は、例えば、図5に示したように、inputポート212a、FIFOメモリ部212b、行先アドレス決定部212c、アービタ212dおよびoutputポート212eを有する。ルータ212のinputポート212aが、ルータ120のoutputポート125に接続される。ルータ212のinputポート212aが、隣接する4つのルータ212のoutputポート212eに接続される。ルータ212のoutputポート212eが、隣接する4つのルータ212のinputポート212aに接続される。ルータ212のoutputポート212eが、共通のコアC内のプロセッサ211に接続される。 FIG. 5 shows an example of a schematic configuration of the router 212. For example, as shown in FIG. 5, the router 212 has an input port 212a, a FIFO memory unit 212b, a destination address determination unit 212c, an arbiter 212d, and an output port 212e. The input port 212a of the router 212 is connected to the output port 125 of the router 120. The input port 212a of the router 212 is connected to the output port 212e of four adjacent routers 212. The output port 212e of the router 212 is connected to the input port 212a of four adjacent routers 212. The output port 212e of the router 212 is connected to the processor 211 in the common core C.
 inputポート212aは、4つのポート(eastIN,southIN,westIN,northIN)と、1つのポート(UpIN)と、1つのポート(DownIN)とを有する。4つのポート(eastIN,southIN,westIN,northIN)は、隣接する4つのルータ212のoutputポート212eに接続される。1つのポート(UpIN)は、ルータ120のoutputポート125に接続される。1つのポート(DownIN)は、共通のコアC内のプロセッサ211に接続される。inputポート212aは、入力された送信データDAをFIFOメモリ部212bに出力する。 The input port 212a has four ports (eastIN, southIN, westIN, northIN), one port (UpIN), and one port (DownIN). The four ports (eastIN, southIN, westIN, northIN) are connected to the output ports 212e of four adjacent routers 212. One port (UpIN) is connected to the output port 125 of the router 120. One port (DownIN) is connected to the processor 211 in the common core C. The input port 212a outputs the input transmission data DA to the FIFO memory unit 212b.
 FIFOメモリ部212bは、inputポート212aから入力された送信データDAを一時保存する。FIFOメモリ部212bは、FIFOメモリ部212bに保存された送信データDAを、アービタ212dの制御に応じて出力する。行先アドレス決定部212cは、ルーティングテーブル(TBL)を有し、TBLに基づいて、送信データDAの送信先のニューロンのアドレスを取得する。行先アドレス決定部212cは、取得したアドレスと画素信号Dpとを対応付けてアービタ212dに出力する。 The FIFO memory unit 212b temporarily stores the transmission data DA input from the input port 212a. The FIFO memory unit 212b outputs the transmission data DA stored in the FIFO memory unit 212b according to the control of the arbiter 212d. The destination address determination unit 212c has a routing table (TBL) and obtains the address of the neuron to which the transmission data DA is to be sent based on the TBL. The destination address determination unit 212c associates the obtained address with the pixel signal Dp and outputs it to the arbiter 212d.
 アービタ212dは、inputポート212aに入力された複数の送信データDAの出力を要求するリクエストを調停し、その調停結果(即ち、送信データDAの出力の許可/不許可)に基づく応答をoutputポート212eに出力する。アービタ212dは、例えば、調停結果に基づいて生成した制御データctl1をルータ120に出力してもよい。制御データctl1には、例えば、コアアレイ部210内の各ニューロンの動作状態が含まれる。outputポート212eは、送信データDAを、隣接する4つのルータ212のinputポート212a、ルータ120のinputポート121、および共通のコアC内のプロセッサ211のいずれかに出力する。 The arbiter 212d arbitrates requests for output of multiple pieces of transmission data DA input to the input port 212a, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of the transmission data DA) to the output port 212e. The arbiter 212d may, for example, output control data ctl1 generated based on the arbitration result to the router 120. The control data ctl1 includes, for example, the operating state of each neuron in the core array unit 210. The output port 212e outputs the transmission data DA to the input ports 212a of the four adjacent routers 212, the input port 121 of the router 120, or the processor 211 in the common core C.
 ルータ212の構成は、図5に示した構成に限定されるものではない。リクエストの調停を行うアービタや、FIFOメモリを用いたバッファリング、TBLを用いた出力先選定などの機能を備えたものであれば、ルータ212の構成は、図5に示した構成に限定されるものではない。 The configuration of the router 212 is not limited to the configuration shown in FIG. 5. The configuration of the router 212 is not limited to the configuration shown in FIG. 5 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
(プロセッサ211)
 図6は、プロセッサ211の機能ブロック例を表したものである。プロセッサ211は、上述したように、SNN・ハードウェアによって構成される。プロセッサ211は、ルータ212から受け取った送信データDAに含まれる画素データDpに対して、SNNを用いた信号処理を行う。プロセッサ211は、例えば、図6に示したように、ニューロンI/O部211a、積和演算部211b、重み格納メモリ部211c、膜電位メモリ部211dおよびLIF(Leaky integrate-and-fire)部21dを有する。プロセッサ211の構成は、図6に示した構成に限定されるものではない。
(Processor 211)
6 shows an example of a functional block of the processor 211. As described above, the processor 211 is configured by SNN hardware. The processor 211 performs signal processing using SNN on pixel data Dp included in the transmission data DA received from the router 212. For example, as shown in FIG. 6, the processor 211 has a neuron I/O unit 211a, a product-sum operation unit 211b, a weight storage memory unit 211c, a membrane potential memory unit 211d, and an LIF (Leaky integrate-and-fire) unit 21d. The configuration of the processor 211 is not limited to the configuration shown in FIG. 6.
 ニューロンI/O部211aは、ルータ212から受け取った送信データDAに含まれる画素データDpをスパイク信号として積和演算部211bに出力する。ニューロンI/O部211aは、ルータ212から受け取った画素データDpを、ルータ212から受け取ったニューロン送信先アドレスと対応付けて積和演算部211bに出力する。積和演算部211bは、ニューロンI/O部211aから入力されるスパイク信号に対して、ニューロン送信先アドレスごとに設定された所定の重み値を掛け、ニューロン送信先アドレスごとに入力スパイク数を加算する積和演算を行う。積和演算部211bは、そのようにして得られた演算結果を、ニューロンI/O部211aを介して膜電位メモリ部211dに格納する。上記の重み値は、重み格納メモリ部211cに格納される。 The neuron I/O unit 211a outputs pixel data Dp contained in the transmission data DA received from the router 212 as a spike signal to the product-sum calculation unit 211b. The neuron I/O unit 211a outputs the pixel data Dp received from the router 212 to the product-sum calculation unit 211b in association with the neuron destination address received from the router 212. The product-sum calculation unit 211b multiplies the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each neuron destination address, and performs a product-sum calculation to add up the number of input spikes for each neuron destination address. The product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a. The weight value is stored in the weight storage memory unit 211c.
 ニューロンI/O部211aは、積和演算部211bから受け取った、ニューロン送信先アドレスごとの値(積和演算の結果)を膜電位として膜電位メモリ部211dに格納する。この膜電位は、いわゆる中間状態と呼ばれるものである。膜電位メモリ部211dにおいて、中間状態がニューロンごとに定義されており、ニューロンI/O部211aを介する積和演算部211bからの入力に基づいて変化する。LIF部211dは、漏れ積分発火処理を行う。LIF部211dは、膜電位メモリ部211dに格納された膜電位に対して所定の膜時定数を掛けることにより、膜電位に対して時間的な変化(漏れ)を生じさせる。LIF部211dは、さらに、中間状態の1つもしくは複数の値が所定の閾値を超えたときにスパイク信号をルータ212に出力する。LIF部211dは、スパイク信号を、中間状態が所定の閾値を超えたニューロンのアドレスと対応付けてルータ212に出力する。 The neuron I/O unit 211a stores the value (result of the product-sum operation) for each neuron destination address received from the product-sum operation unit 211b as a membrane potential in the membrane potential memory unit 211d. This membrane potential is what is called an intermediate state. In the membrane potential memory unit 211d, an intermediate state is defined for each neuron, and changes based on the input from the product-sum operation unit 211b via the neuron I/O unit 211a. The LIF unit 211d performs leaky integration and firing processing. The LIF unit 211d multiplies the membrane potential stored in the membrane potential memory unit 211d by a predetermined membrane time constant, thereby causing a temporal change (leakage) in the membrane potential. The LIF unit 211d further outputs a spike signal to the router 212 when one or more values of the intermediate state exceed a predetermined threshold. The LIF unit 211d outputs the spike signal to the router 212 in association with the address of the neuron whose intermediate state exceeds the predetermined threshold.
[効果]
 次に、撮像装置1000の効果について説明する。
[effect]
Next, the effects of the imaging device 1000 will be described.
 本実施の形態では、画素アレイ部110と各プロセッサ211とが、ルータ120,212を介して直接、接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳(スパイクの宛先が同時期に一か所に集中すること)を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the pixel array unit 110 and each processor 211 are directly connected via routers 120 and 212. This makes it possible to reduce congestion (the destination of spikes concentrating at one location at the same time) in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、ルータ120が複数の受光画素Pもしくは行読み出し回路と複数のプロセッサ211とに接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。なお、行読み出し回路が設けられる場合、画素アレイ部110からデータを出力する際に、全ての受光画素Pからの読み出しを待つ必要がない。そのため、全ての受光画素Pから読み出す場合と比べて、輻輳を低減することができる。 In this embodiment, the router 120 is connected to multiple light receiving pixels P or row readout circuits and multiple processors 211. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN. Note that when a row readout circuit is provided, it is not necessary to wait for readout from all light receiving pixels P when outputting data from the pixel array unit 110. Therefore, it is possible to reduce congestion, compared to when reading out from all light receiving pixels P.
 本実施の形態では、上述のデジタル変換器により得られたデジタル信号が、ルータ120,212を介して複数のプロセッサ211に送信される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the digital signal obtained by the above-mentioned digital converter is transmitted to multiple processors 211 via routers 120 and 212. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to the case where the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、ルータ120,212には、FIFOメモリ122,212bおよびアービタ124,212dが設けられ、FIFOメモリ122,212bに保存された複数のデジタル信号がアービタ124,212dの制御に応じて複数のプロセッサ212に順次送信される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the routers 120, 212 are provided with FIFO memories 122, 212b and arbiters 124, 212d, and multiple digital signals stored in the FIFO memories 122, 212b are sequentially transmitted to multiple processors 212 under the control of the arbiters 124, 212d. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to the conventional case where the pixel array and the multiprocessor are connected via a serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、複数の受光画素Pに対して割り当てられた第1階層のルータ120と、プロセッサ211ごとに1つずつ割り当てられた第2階層の複数のルータ212とが設けられる。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, a first-level router 120 is provided that is assigned to a plurality of light-receiving pixels P, and a plurality of second-level routers 212 are provided, one for each processor 211. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to the case where the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、ルータ120は、各受光画素Pもしくは行読み出し回路と各ルータ212とに接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。なお、行読み出し回路が設けられる場合、画素アレイ部110からデータを出力する際に、全ての受光画素Pからの読み出しを待つ必要がない。そのため、全ての受光画素Pから読み出す場合と比べて、輻輳を低減することができる。 In this embodiment, the router 120 is connected to each light receiving pixel P or row readout circuit and each router 212. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN. Note that when a row readout circuit is provided, it is not necessary to wait for readout from all light receiving pixels P when outputting data from the pixel array unit 110. Therefore, it is possible to reduce congestion, compared to when reading out from all light receiving pixels P.
 本実施の形態では、上述のデジタル変換器の動作状態についてのデータがルータ120によって各ルータ212に送信される。これにより、ルータ120から取得した動作状態についてのデータに基づいて、上述のデジタル変換器により得られたデジタル信号の送信先を各ルータ212によって決定することができる。その結果、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, data on the operating state of the digital converter described above is transmitted by the router 120 to each router 212. This allows each router 212 to determine the destination of the digital signal obtained by the digital converter described above based on the data on the operating state obtained from the router 120. As a result, congestion in data transmission from the pixel array unit 110 to each processor 211 can be reduced compared to when the pixel array and the multiprocessor are connected via a conventional serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
 本実施の形態では、プロセッサ211内のニューロンの動作状態についてのデータが、プロセッサ211に対応するルータ212によって、ルータ120に送信される。これにより、ルータ212からのデータに基づいて、受光画素Pから得られたデータの送信先をルータ120によって決定することができる。その結果、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, data on the operating state of neurons in processor 211 is transmitted to router 120 by router 212 corresponding to processor 211. This allows router 120 to determine the destination of data obtained from light-receiving pixel P based on the data from router 212. As a result, congestion in data transmission from pixel array unit 110 to each processor 211 can be reduced compared to the conventional case where a pixel array and a multiprocessor are connected via a serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
 本実施の形態では、センサチップ1000AとSNNチップ1000Bとが、パッド電極PE1とパッド電極PE2とを互いに重ね合わせて積層される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ110から各プロセッサ211までのデータ伝送距離を短くすることができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other. This makes it possible to shorten the data transmission distance from the pixel array 110 to each processor 211 compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using SNN.
 本実施の形態では、パッド電極PE1は、センサチップ1000Aにおいて、受光面とは反対側の表面に設けられ、上述のデジタル変換器により得られたデジタル信号が出力される配線に接続される。パッド電極PE2は、SNNチップ1000Bの表面に設けられ、ルータ120の入力端に接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ110から各プロセッサ211までのデータ伝送距離を短くすることができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to a wiring that outputs the digital signal obtained by the above-mentioned digital converter. The pad electrode PE2 is provided on the surface of the SNN chip 1000B, and is connected to the input end of the router 120. This makes it possible to shorten the data transmission distance from the pixel array 110 to each processor 211, compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
<2.第1の実施の形態の変形例>
 次に、上記実施の形態に係る撮像装置1000の変形例について説明する。以下では、上記実施の形態と共通の構成要素に対しては、同一の符号を付与し、説明を適宜省略するものとする。
2. Modification of the First Embodiment
Next, a modified example of the imaging device 1000 according to the above embodiment will be described. In the following, the same reference numerals will be given to the components common to the above embodiment, and the description will be omitted as appropriate.
 図7は、図2に示した撮像装置1000の積層構造の一変形例を表したものである。上記実施の形態において、ルータ120が、例えば、図7に示したように二次元配置された複数のルータ120Aによって構成されてもよい。 FIG. 7 shows a modified example of the stacked structure of the imaging device 1000 shown in FIG. 2. In the above embodiment, the router 120 may be configured, for example, by a plurality of routers 120A arranged two-dimensionally as shown in FIG. 7.
 本変形例では、画素アレイ部110において、複数の受光画素Pは複数のグループ(第1のグループ)に分割されており、第1のグループごとに分割された複数の受光画素Pが画素アレイ部110Aを構成する。画素アレイ部110は、二次元配置された複数の画素アレイ部110Aによって構成される。本変形例では、さらに、コアアレイ部210において、複数のコアCは複数のグループ(第2のグループ)に分割されており、第2のグループごとに分割された複数のコアCがコアアレイ部210Aを構成する。コアアレイ部210Aは、二次元配置された複数のコアCによって構成される。 In this modified example, in the pixel array section 110, the multiple light receiving pixels P are divided into multiple groups (first groups), and the multiple light receiving pixels P divided into each first group constitute the pixel array section 110A. The pixel array section 110 is composed of multiple pixel array sections 110A arranged two-dimensionally. In this modified example, further, in the core array section 210, the multiple cores C are divided into multiple groups (second groups), and the multiple cores C divided into each second group constitute the core array section 210A. The core array section 210A is composed of multiple cores C arranged two-dimensionally.
 複数のルータ120Aは、複数の画素アレイ部110Aに対して1つずつ割り当てられており、さらに、複数のコアアレイ部210Aに対して1つずつ割り当てられる。ルータ120Aは、対応する画素アレイ部110A(各受光画素Pもしくは行読み出し回路)に接続されるとともに、対応するコアアレイ部210A(各ルータ212)に接続される。ルータ120Aは、さらに、隣接する複数のルータ120Aに接続される。SNNチップ1000Bにおいて、ルータ120Aは、対応する画素アレイ部110A(複数の受光画素P)が設けられた箇所と対向する箇所に設けられるとともに、対応するコアアレイ部210A(複数のルータ212)が設けられた箇所と対向する箇所に設けられる。画素アレイ部110と各プロセッサ211との間に設けられたルータは、第1階層の複数のルータ120Aと第2階層の複数のルータ212とにより構成された複数階層(2階層)となっている。 The routers 120A are assigned to the pixel array units 110A one by one, and further assigned to the core array units 210A one by one. The routers 120A are connected to the corresponding pixel array units 110A (each light receiving pixel P or row readout circuit) and to the corresponding core array units 210A (each router 212). The routers 120A are further connected to the adjacent routers 120A. In the SNN chip 1000B, the routers 120A are provided at locations opposite to the locations where the corresponding pixel array units 110A (multiple light receiving pixels P) are provided, and are provided at locations opposite to the locations where the corresponding core array units 210A (multiple routers 212) are provided. The routers provided between the pixel array unit 110 and each processor 211 are multiple hierarchical (two hierarchical) layers, consisting of multiple routers 120A in the first layer and multiple routers 212 in the second layer.
 本変形例では、さらに、各ルータ120Aのinputポート121が、隣接する4つのルータ120Aのoutputポート125に接続される。各ルータ120Aのoutputポート125が、隣接する4つのルータ120Aのinputポート121に接続される。 In this modified example, the input port 121 of each router 120A is further connected to the output port 125 of the four adjacent routers 120A. The output port 125 of each router 120A is connected to the input port 121 of the four adjacent routers 120A.
 センサチップ1000Aには、受光画素Pごとに1または複数のパッド電極PE1が設けられる。各パッド電極PE1は、センサチップ1000Aにおいて、受光面とは反対側の表面に設けられ、送信データDAが出力される配線に接続される。SNNチップ1000Bには、受光画素Pごとに1または複数のパッド電極PE2が設けられる。各パッド電極PE2は、SNNチップ1000Bの表面に設けられる。画素アレイ部110Aに設けられた各パッド電極PE2は、画素アレイ部110Aに対応するルータ120Aの入力端に接続される。センサチップ1000AとSNNチップ1000Bとは、パッド電極PE1とパッド電極PE2とを互いに重ね合わせて積層される。 The sensor chip 1000A is provided with one or more pad electrodes PE1 for each light receiving pixel P. Each pad electrode PE1 is provided on the surface of the sensor chip 1000A opposite the light receiving surface, and is connected to a wiring that outputs the transmission data DA. The SNN chip 1000B is provided with one or more pad electrodes PE2 for each light receiving pixel P. Each pad electrode PE2 is provided on the surface of the SNN chip 1000B. Each pad electrode PE2 provided in the pixel array section 110A is connected to the input terminal of the router 120A corresponding to the pixel array section 110A. The sensor chip 1000A and the SNN chip 1000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
 図8は、ルータ120Aの概略構成例を表すものである。ルータ120Aは、例えば、図8に示したように、inputポート121、FIFOメモリ部122、行先アドレス決定部123、アービタ124およびoutputポート125を有する。 FIG. 8 shows an example of the schematic configuration of router 120A. For example, as shown in FIG. 8, router 120A has an input port 121, a FIFO memory unit 122, a destination address determination unit 123, an arbiter 124, and an output port 125.
 inputポート121は、対応する各パッド電極PE2に電気的に接続され、対応する画素アレイ部110Aから送信された複数の画素信号DpをFIFOメモリ部122に出力する。inputポート121は、さらに、隣接する4つのルータ120Aのoutputポート125から送信された複数の画素信号DpをFIFOメモリ部122に出力する。 The input port 121 is electrically connected to each corresponding pad electrode PE2, and outputs a plurality of pixel signals Dp transmitted from the corresponding pixel array unit 110A to the FIFO memory unit 122. The input port 121 further outputs a plurality of pixel signals Dp transmitted from the output ports 125 of the four adjacent routers 120A to the FIFO memory unit 122.
 FIFOメモリ部122は、inputポート121から入力された複数の画素信号Dpを一時保存する。FIFOメモリ部122は、FIFOメモリ部122に保存された複数の画素信号Dpを、アービタ124の制御に応じて順次出力する。行先アドレス決定部123は、ルーティングテーブル(TBL)を有し、TBLに基づいて、各画素データDpの送信先のニューロンのアドレスを取得する。行先アドレス決定部123は、画素信号Dpと、取得したアドレスとを対応付けてアービタ124に出力する。 The FIFO memory unit 122 temporarily stores multiple pixel signals Dp input from the input port 121. The FIFO memory unit 122 sequentially outputs the multiple pixel signals Dp stored in the FIFO memory unit 122 under the control of the arbiter 124. The destination address determination unit 123 has a routing table (TBL) and obtains the address of the neuron to which each piece of pixel data Dp is to be sent based on the TBL. The destination address determination unit 123 associates the pixel signal Dp with the obtained address and outputs it to the arbiter 124.
 アービタ124は、複数の受光画素Pのそれぞれから供給される画素信号Dpの出力を要求するリクエストを調停し、その調停結果(即ち、画素信号Dpの出力の許可/不許可)に基づく応答をoutputポート125に出力する。アービタ124は、例えば、制御データctl1に従って調停を行ってもよい。アービタ124は、例えば、制御データctl1に基づいて、送信データDAの送信先を決定してもよい。 The arbiter 124 arbitrates requests for output of pixel signals Dp supplied from each of the multiple light receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 125. The arbiter 124 may perform arbitration according to control data ctl1, for example. The arbiter 124 may determine the destination of the transmission data DA based on the control data ctl1, for example.
 アービタ124は、例えば、調停結果に基づいて生成した制御データctl2を画素アレイ部110に出力してもよい。アービタ124は、例えば、制御データctl1に基づいて制御データctl2を生成し、生成した制御データctl2を画素アレイ部110に出力してもよい。outputポート125は、送信データDAを各コアCと、隣接するルータ120Aのinputポート121とに送信する。 The arbiter 124 may, for example, output the control data ctl2 generated based on the arbitration result to the pixel array unit 110. The arbiter 124 may, for example, generate the control data ctl2 based on the control data ctl1 and output the generated control data ctl2 to the pixel array unit 110. The output port 125 transmits the transmission data DA to each core C and to the input port 121 of the adjacent router 120A.
 ルータ120Aの構成は、図8に示した構成に限定されるものではない。リクエストの調停を行うアービタや、FIFOメモリを用いたバッファリング、TBLを用いた出力先選定などの機能を備えたものであれば、ルータ120Aの構成は、図8に示した構成に限定されるものではない。 The configuration of router 120A is not limited to the configuration shown in FIG. 8. As long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL, the configuration of router 120A is not limited to the configuration shown in FIG. 8.
 本変形例では、一組の第1のグループおよび第2のグループごとに1つずつ割り当てられた第1階層の複数のルータ120Aと、一組の第1のグループおよび第2のグループごとに複数割り当てられた第2階層の複数のルータ212とが設けられる。これにより、一組の第1のグループおよび第2のグループごとにデータ伝送を行うことができる。その結果、従来のようなシリアライザを介して画素アレイ部とマルチプロセッサ部とが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this modified example, a plurality of routers 120A in the first hierarchical layer are provided, one for each of a set of the first group and the second group, and a plurality of routers 212 in the second hierarchical layer are provided, one for each of a set of the first group and the second group. This allows data transmission for each of a set of the first group and the second group. As a result, congestion in data transmission from the pixel array unit 110 to each processor 211 can be reduced compared to when the pixel array unit and the multiprocessor unit are connected via a serializer as in the conventional case. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
 本変形例では、ルータ120Aが、対応する第1のグループの各受光画素Pと、対応する第2のグループの各ルータ212とに接続され、さらに、対応する第1のグループに隣接する複数のルータ120Aに接続される。各ルータ212が、対応するルータ120Aと、対応するプロセッサ211とに接続され、さらに、隣接する複数のルータ212に接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this modified example, the router 120A is connected to each light receiving pixel P of the corresponding first group and each router 212 of the corresponding second group, and is further connected to multiple routers 120A adjacent to the corresponding first group. Each router 212 is connected to the corresponding router 120A and the corresponding processor 211, and is further connected to multiple adjacent routers 212. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and the multiprocessor are connected via a serializer as in the conventional case. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本変形例では、上述のデジタル変換器の動作状態についてのデータが、ルータ120によって各ルータ212に送信される。ルータ120から取得した、上述のデジタル変換器の動作状態についてのデータに基づいて、上述のデジタル変換器により得られたデジタル信号の送信先が、ルータ212によって決定される。これにより、各プロセッサ211の輻輳の度合に応じたデータ送信を行うことができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this modified example, data on the operating state of the digital converter described above is transmitted by the router 120 to each router 212. Based on the data on the operating state of the digital converter obtained from the router 120, the destination of the digital signal obtained by the digital converter described above is determined by the router 212. This makes it possible to transmit data according to the degree of congestion of each processor 211. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using an SNN.
 本変形例では、プロセッサ211内のニューロンの動作状態についてのデータが、プロセッサ211に対応するルータ212によって、ルータ120Aに送信される。これにより、ルータ212からのデータに基づいて、受光画素Pから得られたデータの送信先をルータ120Aによって決定することができる。その結果、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this modified example, data on the operating state of neurons in processor 211 is transmitted to router 120A by router 212 corresponding to processor 211. This allows router 120A to determine the destination of data obtained from light-receiving pixel P based on the data from router 212. As a result, congestion in data transmission from pixel array unit 110 to each processor 211 can be reduced compared to the conventional case where the pixel array and multiprocessor are connected via a serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using SNN.
<3.第2の実施の形態>
[構成例]
 図9は、本開示の第2の実施の形態に係る撮像装置2000の概略構成例を表すものである。撮像装置2000は、例えば、図9に示したように、センサ部300およびプロセッサ部400を備える。センサ部300は、画素アレイ部310を有する。プロセッサ部400は、コアアレイ部410を有する。コアアレイ部410は、二次元配置された複数のコアCを有する。各コアCは、プロセッサ211およびルータ412を有する。コアアレイ部410は、二次元配置された複数のプロセッサ211と、二次元配置された複数のルータ412とを有する。二次元配置された複数のプロセッサ211が、本開示の一実施の形態に係る「マルチプロセッサ」の一具体例に相当する。プロセッサ211は、SNN・ハードウェアによって構成される。画素アレイ部310と各プロセッサ211とは、複数のルータ412を介して直接、接続される。
3. Second embodiment
[Configuration example]
FIG. 9 shows a schematic configuration example of an imaging device 2000 according to a second embodiment of the present disclosure. For example, as shown in FIG. 9, the imaging device 2000 includes a sensor unit 300 and a processor unit 400. The sensor unit 300 includes a pixel array unit 310. The processor unit 400 includes a core array unit 410. The core array unit 410 includes a plurality of cores C arranged two-dimensionally. Each core C includes a processor 211 and a router 412. The core array unit 410 includes a plurality of processors 211 arranged two-dimensionally and a plurality of routers 412 arranged two-dimensionally. The plurality of processors 211 arranged two-dimensionally correspond to a specific example of a "multiprocessor" according to an embodiment of the present disclosure. The processor 211 is configured by SNN hardware. The pixel array unit 310 and each processor 211 are directly connected via a plurality of routers 412.
 画素アレイ部310は、二次元配置された複数の受光画素Pを有する。受光画素Pは、例えば、CMOS素子、EVS素子、または、SPAD素子を含む。受光画素Pは、外部から入射した光を検出することにより検出信号を生成する。画素アレイ部310は、例えば、生成した検出信号を直接、複数のルータ412に送信してもよい。画素アレイ部310は、例えば、上記検出信号を画素アレイ部110内のADCもしくはカウンタによってデジタル化し、それにより得られたデジタル信号を複数のルータ412に送信してもよい。画素アレイ部310は、受光画素Pでの光検出に基づいて生成されたデータ(例えば、上記検出信号、または、上記デジタル信号)を画素データDpとして複数のルータ412に送信する。上記検出信号、または、上記デジタル信号は、いわゆるスパイク信号に相当する。 The pixel array section 310 has a plurality of light receiving pixels P arranged two-dimensionally. The light receiving pixels P include, for example, a CMOS element, an EVS element, or a SPAD element. The light receiving pixels P generate detection signals by detecting light incident from the outside. The pixel array section 310 may, for example, directly transmit the generated detection signals to the plurality of routers 412. The pixel array section 310 may, for example, digitize the detection signals using an ADC or a counter in the pixel array section 110, and transmit the digital signals thus obtained to the plurality of routers 412. The pixel array section 310 transmits data (for example, the detection signals or the digital signals) generated based on the light detection at the light receiving pixels P as pixel data Dp to the plurality of routers 412. The detection signals or the digital signals correspond to so-called spike signals.
 画素アレイ部310は、複数のルータ412からの制御データctl2に基づいて、各受光画素Pから得られた画素データDpをルータ412に送信してもよい。この制御データctl2には、例えば、画素アレイ部310からのデータ出力の要否についての受光画素Pごとのデータが含まれる。画素アレイ部310は、制御データctl2に基づいて、各受光画素Pから得られた画素データDpの出力の要否を決定し、出力要と決定された受光画素Pの画素データDpをルータ412に送信してもよい。 The pixel array unit 310 may transmit pixel data Dp obtained from each light receiving pixel P to the router 412 based on control data ctl2 from the multiple routers 412. This control data ctl2 includes, for example, data for each light receiving pixel P regarding whether data output from the pixel array unit 310 is required. The pixel array unit 310 may determine whether or not it is required to output the pixel data Dp obtained from each light receiving pixel P based on the control data ctl2, and transmit the pixel data Dp of the light receiving pixel P determined to require output to the router 412.
 ルータ412は、画素アレイ部310から画素データDpを取得すると、取得した画素データDpに対応する画素アドレスを参照して、画素データDpの送信先のニューロンのアドレスを取得する。ルータ412には、ルーティングテーブルが設けられており、ルータ412は、ルーティングテーブルを用いて、画素データDpの送信先のニューロンのアドレスを取得する。ルータ412は、必要に応じて、画素データDpを取得したときの時間データ(タイムスタンプ)を生成してもよい。 When the router 412 acquires pixel data Dp from the pixel array unit 310, it references the pixel address corresponding to the acquired pixel data Dp to acquire the address of the neuron to which the pixel data Dp is to be sent. The router 412 is provided with a routing table, and uses the routing table to acquire the address of the neuron to which the pixel data Dp is to be sent. If necessary, the router 412 may generate time data (timestamp) when the pixel data Dp was acquired.
 ルータ412は、取得したアドレスと画素データDpとを含む送信データDAを送信する。送信先が当該ルータ412に対応するプロセッサ211内のニューロンである場合、ルータ412は、送信データDAを当該ルータ412に対応するプロセッサ211に送信する。送信先が当該ルータ412に隣接するルータ412に対応するプロセッサ211内のニューロンである場合、ルータ412は、送信データDAを、当該ルータ412に隣接するルータ412に対応するプロセッサ211に送信する。プロセッサ211は、ルータ412から取得した送信データDAに含まれる画素データDpに対して、SNNを用いた信号処理を行う。コアアレイ部410は、プロセッサ211における信号処理により得られたデータDoutを外部に出力する。 The router 412 transmits transmission data DA including the acquired address and pixel data Dp. If the destination is a neuron in the processor 211 corresponding to the router 412, the router 412 transmits the transmission data DA to the processor 211 corresponding to the router 412. If the destination is a neuron in the processor 211 corresponding to the router 412 adjacent to the router 412, the router 412 transmits the transmission data DA to the processor 211 corresponding to the router 412 adjacent to the router 412. The processor 211 performs signal processing using SNN on the pixel data Dp included in the transmission data DA acquired from the router 412. The core array unit 410 outputs data Dout obtained by signal processing in the processor 211 to the outside.
 ルータ412は、画素アレイ部110に含まれるデジタル変換器(例えば、読出部113,116,119に含まれるデジタル変換器)の動作状態についてのデータに基づいて、上述のデジタル変換器により得られたデジタル信号の送信先を決定する。この動作状態には、例えば、デジタル変換器の動作の有無や、デジタル変換のビット幅、デジタル変換器の動作タイミングもしくは動作周波数が含まれる。 The router 412 determines the destination of the digital signal obtained by the digital converter included in the pixel array unit 110 (for example, the digital converter included in the readout units 113, 116, and 119) based on data about the operating state of the digital converter. This operating state includes, for example, whether the digital converter is operating, the bit width of the digital conversion, and the operating timing or operating frequency of the digital converter.
 ルータ412は、当該ルータ412に対応するプロセッサ211内のLIF部211dから得られたデータ(スパイク信号およびアドレス)に基づいて、当該ルータ412に対応するプロセッサ211内のニューロンの動作状態についてのデータを生成する。ルータ412は、生成したデータ(当該ルータ412に対応するプロセッサ211内のニューロンの動作状態についてのデータ)に基づいて制御データCtl2を生成し、生成した制御データCtl2を画素アレイ310に送信してもよい。 The router 412 generates data on the operating state of the neuron in the processor 211 corresponding to the router 412 based on the data (spike signal and address) obtained from the LIF unit 211d in the processor 211 corresponding to the router 412. The router 412 may generate control data Ctl2 based on the generated data (data on the operating state of the neuron in the processor 211 corresponding to the router 412) and transmit the generated control data Ctl2 to the pixel array 310.
 図10は、撮像装置2000の積層構造の一例を表したものである。画素アレイ部310は、例えば、図10に示したように、センサチップ2000Aによって構成される。センサチップ2000Aにおいて、画素アレイ部310は半導体基板上に形成される。プロセッサ部400が、例えば、図10に示したように、SNNチップ2000Bによって構成される。SNNチップ2000Bにおいて、プロセッサ部400は半導体基板上に形成される。 FIG. 10 shows an example of a layered structure of the imaging device 2000. The pixel array section 310 is formed, for example, by a sensor chip 2000A as shown in FIG. 10. In the sensor chip 2000A, the pixel array section 310 is formed on a semiconductor substrate. The processor section 400 is formed, for example, by an SNN chip 2000B as shown in FIG. 10. In the SNN chip 2000B, the processor section 400 is formed on a semiconductor substrate.
 センサチップ2000Aには、受光画素Pごとに1または複数のパッド電極PE1が設けられる。各パッド電極PE1は、センサチップ2000Aにおいて、受光面とは反対側の表面に設けられ、画素データDpが出力される配線に接続される。SNNチップ2000Bには、受光画素Pごとに1または複数のパッド電極PE2が設けられる。各パッド電極PE2は、SNNチップ2000Bの表面に設けられ、ルータ412の入力端に接続される。センサチップ2000AとSNNチップ2000Bとは、パッド電極PE1とパッド電極PE2とを互いに重ね合わせて積層される。 The sensor chip 2000A is provided with one or more pad electrodes PE1 for each light receiving pixel P. Each pad electrode PE1 is provided on the surface of the sensor chip 2000A opposite the light receiving surface, and is connected to wiring that outputs pixel data Dp. The SNN chip 2000B is provided with one or more pad electrodes PE2 for each light receiving pixel P. Each pad electrode PE2 is provided on the surface of the SNN chip 2000B, and is connected to the input terminal of the router 412. The sensor chip 2000A and the SNN chip 2000B are stacked with the pad electrodes PE1 and PE2 overlapping each other.
 画素アレイ部310において、複数の受光画素Pは複数のグループ(第1のグループ)に分割されており、第1のグループごとに分割された複数の受光画素Pが画素アレイ部310Aを構成する。画素アレイ部310は、二次元配置された複数の画素アレイ部310Aによって構成される。コアアレイ部410において、複数のルータ412が二次元配置されるとともに、複数のプロセッサ211が二次元配置される。 In the pixel array section 310, the multiple light receiving pixels P are divided into multiple groups (first groups), and the multiple light receiving pixels P divided into each first group constitute the pixel array section 310A. The pixel array section 310 is composed of multiple pixel array sections 310A arranged two-dimensionally. In the core array section 410, multiple routers 412 are arranged two-dimensionally, and multiple processors 211 are arranged two-dimensionally.
 複数のルータ412は、複数の画素アレイ部310Aに対して1つずつ割り当てられており、さらに、複数のプロセッサ211に対して1つずつ割り当てられる。ルータ412は、対応する画素アレイ部310A(各受光画素P)に接続されるとともに、対応するプロセッサ211に接続される。ルータ412は、さらに、隣接する複数のルータ412に接続される。SNNチップ2000Bにおいて、ルータ412は、対応する画素アレイ部310A(複数の受光画素P)が設けられた箇所と対向する箇所に設けられるとともに、対応するプロセッサ211が設けられた箇所と対向する箇所に設けられる。SNNチップ2000Bにおいて、ルータ412は、対応するプロセッサ211が設けられた箇所に隣接する箇所と対向する箇所に設けられてもよい。画素アレイ部310と各プロセッサ211との間に設けられたルータは、複数のルータ412により構成された単一階層(1階層)となっている。 The routers 412 are assigned one by one to the pixel array units 310A, and further assigned one by one to the processors 211. The router 412 is connected to the corresponding pixel array unit 310A (each light receiving pixel P) and to the corresponding processor 211. The router 412 is further connected to adjacent routers 412. In the SNN chip 2000B, the router 412 is provided at a location opposite the location where the corresponding pixel array unit 310A (multiple light receiving pixels P) is provided, and is provided at a location opposite the location where the corresponding processor 211 is provided. In the SNN chip 2000B, the router 412 may be provided at a location opposite the location adjacent to the location where the corresponding processor 211 is provided. The router provided between the pixel array unit 310 and each processor 211 is a single layer (one layer) composed of multiple routers 412.
 本変形例では、さらに、各ルータ412のinputポート412aが、隣接する4つのルータ412aのoutputポート412eに接続される。各ルータ412のoutputポート412eが、隣接する4つのルータ412のinputポート412aに接続される。 In this modified example, the input port 412a of each router 412 is further connected to the output port 412e of the four adjacent routers 412a. The output port 412e of each router 412 is connected to the input port 412a of the four adjacent routers 412.
 図11は、ルータ412の概略構成例を表すものである。ルータ412は、例えば、図11に示したように、inputポート412a、FIFOメモリ部412b、行先アドレス決定部412c、アービタ412dおよびoutputポート412eを有する。ルータ412のinputポート412aが、対応する画素アレイ310Aの各受光画素P(もしくは行読み出し回路)に接続される。ルータ412のinputポート412aが、隣接する4つのルータ412のoutputポート412eに接続される。ルータ412のoutputポート412eが、隣接する4つのルータ412のinputポート412aに接続される。ルータ412のoutputポート412eが、共通のコアC内のプロセッサ211に接続される。 FIG. 11 shows an example of a schematic configuration of the router 412. For example, as shown in FIG. 11, the router 412 has an input port 412a, a FIFO memory unit 412b, a destination address determination unit 412c, an arbiter 412d, and an output port 412e. The input port 412a of the router 412 is connected to each light receiving pixel P (or row readout circuit) of the corresponding pixel array 310A. The input port 412a of the router 412 is connected to the output port 412e of four adjacent routers 412. The output port 412e of the router 412 is connected to the input port 412a of four adjacent routers 412. The output port 412e of the router 412 is connected to the processor 211 in the common core C.
 inputポート412aは、4つのポート(eastIN,southIN,westIN,northIN)と、1つのポート(PxIN)と、1つのポート(localIN)とを有する。4つのポート(eastIN,southIN,westIN,northIN)は、隣接する4つのルータ412のoutputポート412eに接続される。1つのポート(PxIN)は、対応する画素アレイ部310Aに接続される。1つのポート(localIN)は、共通のコアC内のプロセッサ211に接続される。inputポート412aは、入力された画素データDpをFIFOメモリ部412bに出力する。 The input port 412a has four ports (eastIN, southIN, westIN, northIN), one port (PxIN), and one port (localIN). The four ports (eastIN, southIN, westIN, northIN) are connected to the output ports 412e of the four adjacent routers 412. One port (PxIN) is connected to the corresponding pixel array unit 310A. One port (localIN) is connected to the processor 211 in the common core C. The input port 412a outputs the input pixel data Dp to the FIFO memory unit 412b.
 FIFOメモリ部412bは、inputポート412aから入力された複数の画素信号Dpを一時保存する。FIFOメモリ部412bは、FIFOメモリ部412bに保存された複数の画素信号Dpを、アービタ412dの制御に応じて出力する。行先アドレス決定部412cは、ルーティングテーブル(TBL)を有し、TBLに基づいて、各画素データDpの送信先のニューロンのアドレスを取得する。行先アドレス決定部412cは、取得したアドレスと画素信号Dpとを対応付けてアービタ412dに出力する。 The FIFO memory unit 412b temporarily stores multiple pixel signals Dp input from the input port 412a. The FIFO memory unit 412b outputs the multiple pixel signals Dp stored in the FIFO memory unit 412b according to the control of the arbiter 412d. The destination address determination unit 412c has a routing table (TBL) and obtains the address of the neuron to which each pixel data Dp is to be sent based on the TBL. The destination address determination unit 412c associates the obtained address with the pixel signal Dp and outputs it to the arbiter 412d.
 アービタ412dは、複数の受光画素Pのそれぞれから供給される画素信号Dpの出力を要求するリクエストを調停し、その調停結果(即ち、画素信号Dpの出力の許可/不許可)に基づく応答をoutputポート412eに出力する。アービタ412dは、例えば、調停結果に基づいて生成した制御データctl1を、隣接するルータ412のアービタ412dに出力してもよい。制御データctl1には、例えば、コアアレイ部410内の各ニューロンの動作状態が含まれる。outputポート412eは、画素信号Dpを、隣接する4つのルータ412のinputポート412aおよび共通のコアC内のプロセッサ211のいずれかに出力する。 The arbiter 412d arbitrates requests for output of pixel signals Dp supplied from each of the multiple light-receiving pixels P, and outputs a response based on the arbitration result (i.e., permission/prohibition of output of pixel signals Dp) to the output port 412e. The arbiter 412d may, for example, output control data ctl1 generated based on the arbitration result to the arbiter 412d of an adjacent router 412. The control data ctl1 includes, for example, the operating state of each neuron in the core array unit 410. The output port 412e outputs the pixel signal Dp to either the input port 412a of the four adjacent routers 412 or the processor 211 in the common core C.
 ルータ412の構成は、図11に示した構成に限定されるものではない。リクエストの調停を行うアービタや、FIFOメモリを用いたバッファリング、TBLを用いた出力先選定などの機能を備えたものであれば、ルータ412の構成は、図11に示した構成に限定されるものではない。 The configuration of the router 412 is not limited to the configuration shown in FIG. 11. The configuration of the router 412 is not limited to the configuration shown in FIG. 11 as long as it has functions such as an arbiter that arbitrates requests, buffering using FIFO memory, and output destination selection using a TBL.
 図12は、プロセッサ211の機能ブロック例を表したものである。プロセッサ211は、上述したように、SNN・ハードウェアによって構成される。プロセッサ211は、ルータ412から受け取った画素信号Dpに対して、SNNを用いた信号処理を行う。プロセッサ211は、例えば、図12に示したように、ニューロンI/O部211a、積和演算部211b、重み格納メモリ部211c、膜電位メモリ部211dおよびLIF部211dを有する。プロセッサ211の構成は、図12に示した構成に限定されるものではない。 FIG. 12 shows an example of a functional block of the processor 211. As described above, the processor 211 is composed of SNN hardware. The processor 211 performs signal processing using SNN on the pixel signal Dp received from the router 412. For example, as shown in FIG. 12, the processor 211 has a neuron I/O unit 211a, a product-sum operation unit 211b, a weight storage memory unit 211c, a membrane potential memory unit 211d, and an LIF unit 211d. The configuration of the processor 211 is not limited to the configuration shown in FIG. 12.
 ニューロンI/O部211aは、ルータ412から受け取った画素信号Dpをスパイク信号として積和演算部211bに出力する。積和演算部211bは、ニューロンI/O部211aから入力されるスパイク信号に対して、ニューロン送信先アドレスごとに設定された所定の重み値を掛け、ニューロン送信先アドレスごとに入力スパイク数を加算する積和演算を行う。積和演算部211bは、そのようにして得られた演算結果を、ニューロンI/O部211aを介して膜電位メモリ部211dに格納する。上記の重み値は、重み格納メモリ部211cに格納される。 The neuron I/O unit 211a outputs the pixel signal Dp received from the router 412 to the product-sum calculation unit 211b as a spike signal. The product-sum calculation unit 211b performs a product-sum calculation by multiplying the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each neuron destination address and adding up the number of input spikes for each neuron destination address. The product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a. The above weight values are stored in the weight storage memory unit 211c.
 ニューロンI/O部211aは、積和演算部211bから受け取った、ニューロン送信先アドレスごとの値(積和演算の結果)を膜電位として膜電位メモリ部211dに格納する。この膜電位は、いわゆる中間状態と呼ばれるものである。LIF部211dは、漏れ積分発火処理を行う。LIF部211dは、膜電位メモリ部211dに格納された膜電位に対して所定の膜時定数を掛けることにより、膜電位に対して時間的な変化(漏れ)を生じさせる。LIF部211dは、さらに、中間状態の1つもしくは複数の値が所定の閾値を超えたときにスパイク信号をルータ412に出力する。 The neuron I/O unit 211a receives the value (result of the product-sum operation) for each neuron destination address from the product-sum operation unit 211b and stores it as a membrane potential in the membrane potential memory unit 211d. This membrane potential is what is called an intermediate state. The LIF unit 211d performs leaky integration and firing processing. The LIF unit 211d multiplies the membrane potential stored in the membrane potential memory unit 211d by a predetermined membrane time constant, thereby causing a temporal change (leakage) in the membrane potential. The LIF unit 211d further outputs a spike signal to the router 412 when one or more values of the intermediate state exceed a predetermined threshold.
[効果]
 次に、撮像装置2000の効果について説明する。
[effect]
Next, the effects of the imaging device 2000 will be described.
 本実施の形態では、ルータ412がプロセッサ211ごとに1つずつ割り当てられる。これにより、画素アレイ部310で得られた画素信号Dpが、ルータ412を介してプロセッサ211に送信される。その結果、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部310から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, one router 412 is assigned to each processor 211. As a result, the pixel signal Dp obtained in the pixel array unit 310 is transmitted to the processor 211 via the router 412. As a result, congestion in data transmission from the pixel array unit 310 to each processor 211 can be reduced compared to when the pixel array and the multiprocessor are connected via a conventional serializer. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、ルータ412が、対応する画素アレイ310Aの各受光画素Pもしくは行読み出し回路と、対応する各プロセッサ211とに接続され、さらに、隣接する複数のルータ412に接続される。これにより、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部110から各プロセッサ211へのデータ送信における輻輳を低減することができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the router 412 is connected to each light receiving pixel P or row readout circuit of the corresponding pixel array 310A and each corresponding processor 211, and is further connected to multiple adjacent routers 412. This makes it possible to reduce congestion in data transmission from the pixel array unit 110 to each processor 211, compared to when the pixel array and the multiprocessor are connected via a conventional serializer. As a result, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
 本実施の形態では、上述のデジタル変換器の動作状態についてのデータに基づいて、上述のデジタル変換器により得られたデジタル信号の送信先が、ルータ412によって決定される。これにより、各プロセッサ211の輻輳の度合に応じたデータ送信を行うことができる。その結果、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, the destination of the digital signal obtained by the digital converter is determined by the router 412 based on data about the operating state of the digital converter. This allows data transmission according to the degree of congestion of each processor 211. As a result, it is possible to achieve a further improvement in the processing speed in signal processing using SNN.
 本実施の形態では、プロセッサ211内のニューロンの動作状態についてのデータに基づいて制御信号ctl1が生成され、生成された制御信号ctl1がルータ412から画素アレイ部310に送信される。これにより、ルータ412からの制御信号ctl1に基づいて、各受光画素Pからのデータ出力の要否を決定し、出力要と決定した受光画素Pから得られたデータをルータ412に送信することができる。その結果、従来のようなシリアライザを介して画素アレイとマルチプロセッサとが接続される場合と比べて、画素アレイ部310から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this embodiment, a control signal ctl1 is generated based on data about the operating state of neurons in the processor 211, and the generated control signal ctl1 is transmitted from the router 412 to the pixel array unit 310. As a result, it is possible to determine whether or not data output is required from each light-receiving pixel P based on the control signal ctl1 from the router 412, and transmit data obtained from light-receiving pixels P that have been determined to require output to the router 412. As a result, congestion in data transmission from the pixel array unit 310 to each processor 211 can be reduced compared to when the pixel array and multiprocessor are connected via a conventional serializer. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
<4.各実施の形態に共通の変形例>
 次に、上記第1の実施の形態およびその変形例に係る撮像装置1000や、上記第2の実施の形態に係る撮像装置2000の共通の変形例について説明する。以下では、上記各実施の形態と共通の構成要素に対しては、同一の符号を付与し、説明を適宜省略するものとする。
4. Modifications common to all embodiments
Next, common modifications of the imaging device 1000 according to the first embodiment and its modifications and the imaging device 2000 according to the second embodiment will be described. In the following, the same reference numerals will be given to components common to the above-mentioned embodiments, and the description will be omitted as appropriate.
[変形例A]
 図13は、画素アレイ部110,110A,310Aの機能ブロック例を表したものである。上記各実施の形態およびそれらの変形例において、画素アレイ部110,110A,310Aは、例えば、図13に示したように、画素アレイ回路101、垂直走査回路102および行読み出し回路103を有してもよい。
[Variation A]
Fig. 13 shows an example of functional blocks of the pixel array units 110, 110A, and 310A. In each of the above-described embodiments and their modified examples, the pixel array units 110, 110A, and 310A may include, for example, a pixel array circuit 101, a vertical scanning circuit 102, and a row readout circuit 103, as shown in Fig. 13.
 画素アレイ回路101は、例えば、図13に示したように、行列状に二次元配置された複数の受光画素Pを有する。画素アレイ回路101には、例えば、画素列ごとに垂直信号線が配置され、画素行ごとに水平信号線が配置される。 The pixel array circuit 101 has a plurality of light-receiving pixels P arranged two-dimensionally in a matrix, for example, as shown in FIG. 13. In the pixel array circuit 101, for example, a vertical signal line is arranged for each pixel column, and a horizontal signal line is arranged for each pixel row.
 垂直走査回路102は、複数の水平信号線を介して複数の受光画素Pを行ごとに選択し、一行分の各受光画素Pで生成された信号を、複数の垂直信号線を介して行読み出し回路103に出力させる。行読み出し回路103は、例えば、図13に示したように、ADC104および水平走査回路105を有する。ADC104は、垂直信号線ごとに1つずつ設けられた複数のADC104aを有し、各ADC104aは、垂直信号線を介して受光画素Pから取得した信号をデジタル変換する。水平走査回路105は、各ADC104aから得られた複数のデジタル信号(画素データDp)を画素行ごとに順次出力する。水平走査回路105は、一行分の複数の画素データDpを含むラスターデータをルータ120,120A,412に出力する。 The vertical scanning circuit 102 selects a number of light receiving pixels P for each row via a number of horizontal signal lines, and outputs the signals generated by each light receiving pixel P for one row to the row readout circuit 103 via a number of vertical signal lines. The row readout circuit 103 has an ADC 104 and a horizontal scanning circuit 105, for example, as shown in FIG. 13. The ADC 104 has a number of ADCs 104a, one for each vertical signal line, and each ADC 104a digitally converts the signals acquired from the light receiving pixels P via the vertical signal line. The horizontal scanning circuit 105 sequentially outputs a number of digital signals (pixel data Dp) obtained from each ADC 104a for each pixel row. The horizontal scanning circuit 105 outputs raster data including a number of pixel data Dp for one row to the routers 120, 120A, and 412.
 各ルータ212,412は、例えば、ルータ120,120A,310Aから得られたデータを所定の通信規格で、対応するプロセッサ211に送信する。各ルータ212,412は、例えば、ルータ120,120A,310Aから得られた1フレーム分(全ての受光画素P)の複数の画素データDpを、フレームスタート(FS)およびフレームエンド(FE)とともに複数のプロセッサ211に送信する。各ルータ212,412は、例えば、FS、1フレーム分(全ての受光画素P)の複数の画素データDp、FEの順に出力する。 Each router 212, 412 transmits data obtained from routers 120, 120A, 310A, for example, to the corresponding processor 211 using a specified communication standard. Each router 212, 412 transmits, for example, a plurality of pixel data Dp for one frame (all light-receiving pixels P) obtained from routers 120, 120A, 310A, to a plurality of processors 211 together with a frame start (FS) and a frame end (FE). Each router 212, 412 outputs, for example, FS, a plurality of pixel data Dp for one frame (all light-receiving pixels P), and FE in that order.
 各プロセッサ211において、LIF部211dは、FSからFEまでの間、漏れ積分発火処理において、中間状態の経時的な漏れ(デクリメント)を無効化する。この無効化には、各行のデータが負性(負のシナプス重み)接続される場合に行うデクリメントは含まれない。 In each processor 211, the LIF unit 211d disables leakage (decrement) over time in the intermediate state during the leaky integrate and fire process from FS to FE. This disablement does not include the decrement that is performed when the data in each row is negatively connected (negative synaptic weight).
 1フレーム分の複数の画素データDpが、行読み出し回路103によって一画素行ごとに順次出力された場合に、プロセッサ211の各ニューロンにおいて、1フレーム分の複数の画素データDpの処理が1画素行ごとに順次行われたとする。このとき、各ニューロンでの処理時刻が1画素行ごと異なってしまう。そこで、LIF部211dは、例えば、図14に示したように、FSからFEまでの期間、デクリメントを無効にしてもよい。図14には、FSからFEまでの期間がデクリメント無効期間ΔXと表現されている。これにより、各ニューロンにおいて、1フレーム分の画素データDpを同一時刻で処理することができる。 When multiple pixel data Dp for one frame are output sequentially by the row readout circuit 103 for each pixel row, the multiple pixel data Dp for one frame are processed sequentially for each pixel row in each neuron of the processor 211. In this case, the processing time in each neuron will be different for each pixel row. Therefore, the LIF unit 211d may disable decrement for the period from FS to FE, for example, as shown in FIG. 14. In FIG. 14, the period from FS to FE is expressed as a decrement disabled period ΔX. This allows each neuron to process one frame of pixel data Dp at the same time.
 図15は、データ伝送の一例を表したものである。各ルータ120,120Aは、例えば、画素アレイ部110,110Aにおける各受光画素Pから得られた複数の画素データDpを所定の通信規格で、対応するルータ212に送信する。各ルータ120,120Aは、例えば、画素アレイ部110,110Aにおける各受光画素Pから得られた複数の画素データDpを、フレームスタート(FS)およびフレームエンド(FE)とともに、対応するルータ212に送信する。各ルータ120,120Aは、例えば、図15に示したように、FS、複数のラスターデータ、およびFEをルータ212に順次送信してもよい。 FIG. 15 shows an example of data transmission. Each router 120, 120A transmits, for example, a plurality of pixel data Dp obtained from each light receiving pixel P in the pixel array unit 110, 110A to the corresponding router 212 using a predetermined communication standard. Each router 120, 120A transmits, for example, a plurality of pixel data Dp obtained from each light receiving pixel P in the pixel array unit 110, 110A to the corresponding router 212 together with a frame start (FS) and a frame end (FE). Each router 120, 120A may, for example, sequentially transmit an FS, a plurality of raster data, and an FE to the router 212 as shown in FIG. 15.
 このとき、ルータ212は、例えば、TBLに基づいて、ルータ120,120Aから取得したデータの送信先のニューロンのアドレスを取得してもよい。続いて、ルータ212は、例えば、図15に示したように、取得したアドレスを、FS、1フレーム分(全ての受光画素P)の複数の画素データDpおよびFEと対応付けて、プロセッサ211に送信してもよい。 At this time, the router 212 may obtain the address of the neuron to which the data obtained from the routers 120 and 120A is to be transmitted, for example, based on the TBL. Next, the router 212 may transmit the obtained address to the processor 211, as shown in FIG. 15, by associating it with FS and a plurality of pixel data Dp and FE for one frame (all light receiving pixels P).
 図16は、データ伝送の一例を表したものである。各ルータ120,120Aは、例えば、図16に示したように、FSおよび1行目のラスターデータ、2行目から(最終行-1)行目までの各行のラスターデータ、ならびに、最終行のラスターデータおよびFEをルータ212に順次送信してもよい。このとき、ルータ212は、例えば、TBLに基づいて、ルータ120,120Aから取得したデータの送信先のニューロンのアドレスを取得してもよい。続いて、ルータ212は、例えば、図16に示したように、取得したアドレスを、FSおよび1画素行分の各画素データDp、2行目から(最終行-1)行目までの各画素データDp、ならびに、FEおよび最終行分の各画素データDpと対応付けて、プロセッサ211に送信してもよい。 FIG. 16 shows an example of data transmission. For example, as shown in FIG. 16, each router 120, 120A may sequentially transmit FS and the raster data of the first row, the raster data of each row from the second row to the (last row - 1) row, and the raster data of the last row and FE to the router 212. At this time, the router 212 may obtain the address of the neuron to which the data obtained from the router 120, 120A is to be transmitted, for example, based on the TBL. Next, the router 212 may transmit the obtained address to the processor 211, as shown in FIG. 16, by associating it with FS and each pixel data Dp of one pixel row, each pixel data Dp of the second row to the (last row - 1) row, and FE and each pixel data Dp of the last row.
 プロセッサ211は、例えば、図15もしくは図16に示したようなデータ形式でFS、1フレーム分の各画素データDp、およびFEを取得すると、FSを取得してからFEを取得するまでの間(デクリメント無効期間ΔX)、デクリメントを無効化する。これにより、各プロセッサ211において、1フレーム分の複数の画素データDpを同一時刻で処理することができる。 When the processor 211 acquires FS, one frame's worth of pixel data Dp, and FE in the data format shown in FIG. 15 or FIG. 16, for example, it disables decrement during the period from when FS is acquired until when FE is acquired (decrement disabled period ΔX). This allows each processor 211 to process one frame's worth of multiple pixel data Dp at the same time.
[変形例B]
 図17は、プロセッサ部200の機能ブロックの一変形例を表したものである。上記各実施の形態およびそれらの変形例において、プロセッサ部200は、例えば、図17に示したように、コアアレイ部210、GlobalFS配信部220およびGlobalFE配信部230を有してもよい。
[Variation B]
17 shows a modified example of the functional blocks of the processor unit 200. In the above-described embodiments and the modified examples thereof, the processor unit 200 may have, for example, a core array unit 210, a GlobalFS distribution unit 220, and a GlobalFE distribution unit 230, as shown in FIG.
 GlobalFS配信部220は、全てのプロセッサ211に対してFSを送信する。GlobalFS配信部220は、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力が開始される前に(つまり複数のプロセッサ211によるデータ読み出し開始前に)、全てのプロセッサ211に対してFSを送信する。GlobalFS配信部220は、例えば、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力開始を示す信号(以下、「入力開始信号」と称する。)をルータ120,120Aから取得すると、全てのプロセッサ211に対してFSを送信する。ルータ120,120Aは、例えば、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力を開始する直前に、入力開始信号をGlobalFS配信部220に送信する。 The GlobalFS distribution unit 220 transmits an FS to all processors 211. The GlobalFS distribution unit 220 transmits an FS to all processors 211 before the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 starts (i.e., before the processors 211 start reading data). For example, when the GlobalFS distribution unit 220 receives a signal (hereinafter referred to as an "input start signal") indicating the start of input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 from the routers 120, 120A, the GlobalFS distribution unit 220 transmits an FS to all processors 211. The routers 120, 120A transmit the input start signal to the GlobalFS distribution unit 220, for example, immediately before the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 starts.
 GlobalFE配信部230は、全てのプロセッサ211に対してFEを送信する。GlobalFE配信部230は、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力が完了した後に(つまり複数のプロセッサ211によるデータ読み出し完了後に)、全てのプロセッサ211に対してFEを送信する。GlobalFE配信部230は、例えば、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力完了を示す信号(以下、「入力完了信号」と称する。)をルータ120,120Aから取得すると、全てのプロセッサ211に対してFEを送信する。ルータ120,120Aは、例えば、1フレーム分(全ての受光画素P)の複数の画素データDpの、複数のプロセッサ211への入力が完了すると同時に(または完了した直後に)、入力完了信号をGlobalFE配信部230に送信する。 The GlobalFE distribution unit 230 transmits an FE to all processors 211. The GlobalFE distribution unit 230 transmits an FE to all processors 211 after the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 is completed (i.e., after the data is read by the processors 211 is completed). For example, when the GlobalFE distribution unit 230 receives a signal indicating the input completion of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 (hereinafter referred to as the "input completion signal") from the router 120, 120A, the GlobalFE distribution unit 230 transmits an FE to all processors 211. The router 120, 120A transmits the input completion signal to the GlobalFE distribution unit 230 at the same time (or immediately after) the input of the pixel data Dp of one frame (all light receiving pixels P) to the processors 211 is completed.
 本変形例では、GlobalFS配信部220およびGlobalFE配信部230を設けることにより、上記変形例Aと同様、各プロセッサ211において、1フレーム分の画素データDpを同一時刻で処理することができる。 In this modified example, by providing a GlobalFS distribution unit 220 and a GlobalFE distribution unit 230, one frame's worth of pixel data Dp can be processed at the same time in each processor 211, as in modified example A above.
[変形例C]
 図18は、図8に示したルータ120Aの機能ブロックの一変形例を表したものである。図19は、図11に示したルータ412の機能ブロックの一変形例を表したものである。画素アレイ部110,310が複数の画素アレイ部110A,310Aによって構成され、画素アレイ部110A,310Aから得られた画素データDpがinputポート121,412aに入力されるとする。このとき、本変形例では、画素アレイ部110A,310Aから入力された画素データDpが、FIFOメモリ部122,412bを介さずに、直接、行先アドレス決定部123,412cに入力される。
[Variation C]
Fig. 18 shows a modified example of the functional blocks of the router 120A shown in Fig. 8. Fig. 19 shows a modified example of the functional blocks of the router 412 shown in Fig. 11. Assume that the pixel array unit 110, 310 is composed of a plurality of pixel array units 110A, 310A, and pixel data Dp obtained from the pixel array unit 110A, 310A is input to the input port 121, 412a. At this time, in this modified example, the pixel data Dp input from the pixel array unit 110A, 310A is input directly to the destination address determination unit 123, 412c without passing through the FIFO memory unit 122, 412b.
 行先アドレス決定部123,412cは、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gに基づいて、画素アレイ部110A,310Aから得られた画素データDpの出力の可否を判定する。行先アドレス決定部123,412cは、例えば、ルーティングテーブル(TBL)に基づいて取得したアドレスと、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gとを対比する。伝送禁止データ127,412gは、例えば、図20(A)に示したように、アドレスごとに伝送可を示す「1」か、伝送禁止を示す「0」が対応付けられたデータである。 The destination address determination unit 123, 412c determines whether or not to output the pixel data Dp obtained from the pixel array unit 110A, 310A based on the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f. The destination address determination unit 123, 412c, for example, compares the address obtained based on the routing table (TBL) with the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f. The transmission prohibition data 127, 412g is data in which, for example, "1" indicating transmission possible or "0" indicating transmission prohibited is associated with each address, as shown in FIG. 20(A).
 伝送禁止制御部126,412fは、当該ルータ212,412に対応するプロセッサ211内のニューロンの動作状態についてのデータに基づいて、伝送禁止データ127,412gを生成する。伝送禁止制御部126,412fは、生成した伝送禁止データ127,412gを所定のメモリに格納する。 The transmission prohibition control unit 126, 412f generates transmission prohibition data 127, 412g based on data about the operating state of the neuron in the processor 211 corresponding to the router 212, 412. The transmission prohibition control unit 126, 412f stores the generated transmission prohibition data 127, 412g in a specified memory.
 ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ127,412gにおいて伝送禁止に該当する場合、行先アドレス決定部123,412cは、画素アレイ部110A,310Aから得られた画素データDpの出力を禁止する。一方、ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ127,412gにおいて伝送可に該当する場合、行先アドレス決定部123,412cは、画素アレイ部110A,310Aから得られた画素データDpをアービタ124,412dに出力する。行先アドレス決定部123,412cは、例えば、図20(A)、図20(B)に示したように、伝送禁止データ127,412gの「1」に対応するアドレスに対しての出力を許可し、伝送禁止データ127,412gの「0」に対応するアドレスに対しての出力を禁止する。 If the address obtained based on the routing table (TBL) corresponds to transmission prohibited in the transmission prohibited data 127, 412g, the destination address determination unit 123, 412c prohibits the output of pixel data Dp obtained from the pixel array unit 110A, 310A. On the other hand, if the address obtained based on the routing table (TBL) corresponds to transmission permitted in the transmission prohibited data 127, 412g, the destination address determination unit 123, 412c outputs the pixel data Dp obtained from the pixel array unit 110A, 310A to the arbiter 124, 412d. For example, as shown in Figures 20(A) and 20(B), the destination address determination unit 123, 412c permits output to addresses corresponding to "1" in the transmission prohibited data 127, 412g and prohibits output to addresses corresponding to "0" in the transmission prohibited data 127, 412g.
 このように、本変形例では、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gに基づいて、画素アレイ部110A,310Aから得られた画素データDpの出力の可否が判定される。これにより、不要なデータの出力をなくすことができるので、画素アレイ部110A,310Aから各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this manner, in this modified example, a determination is made as to whether or not to output pixel data Dp obtained from the pixel array units 110A, 310A based on the transmission prohibition data 127, 412g obtained from the transmission prohibition control units 126, 412f. This makes it possible to eliminate the output of unnecessary data, thereby reducing congestion in data transmission from the pixel array units 110A, 310A to each processor 211. Therefore, a further improvement in processing speed can be achieved in signal processing using an SNN.
 本変形例において、伝送禁止制御部126,412fは、所定の時間が経過した後、伝送禁止データ127,412gをリセットしてもよい。 In this modified example, the transmission prohibition control unit 126, 412f may reset the transmission prohibition data 127, 412g after a predetermined time has elapsed.
 また、本変形例において、ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ127,412gにおいて伝送禁止に該当する場合、行先アドレス決定部123,412cは、画素アレイ部110A,310Aから得られた画素データDpの出力先を、伝送禁止データ127,412gにおいて伝送可に該当するアドレスのニューロンに迂回してもよい。この迂回先のニューロンは、伝送禁止となっているニューロンに対応するアテンションデータを補助的に表現する機能を有する。プロセッサ211は、伝送禁止となっているニューロンが示すデータと、この迂回先のニューロンが示すデータ(アテンションデータ)とを対応付けて出力してもよい。このようにした場合には、新たな情報処理を提供することが可能となる。 Furthermore, in this modified example, if the address obtained based on the routing table (TBL) corresponds to transmission prohibited in the transmission prohibited data 127, 412g, the destination address determination unit 123, 412c may redirect the output destination of the pixel data Dp obtained from the pixel array unit 110A, 310A to a neuron having an address corresponding to transmission permitted in the transmission prohibited data 127, 412g. This detouring neuron has a function of auxiliary representation of attention data corresponding to the neuron for which transmission is prohibited. The processor 211 may output the data indicated by the neuron for which transmission is prohibited and the data indicated by the detouring neuron (attention data) in association with each other. In this case, it becomes possible to provide new information processing.
[変形例D]
 図21は、図18に示したルータ120Aの機能ブロックの一変形例を表したものである。図22は、図19に示したルータ412の機能ブロックの一変形例を表したものである。上記変形例Cにおいて、伝送禁止制御部126,412fは、例えば、図21、図22に示したように、伝送禁止データ127,412gに基づいて生成した制御信号ctl3を垂直走査仮102および水平走査回路105の少なくとも一方に送信してもよい。
[Modification D]
Fig. 21 shows a modified example of the functional blocks of the router 120A shown in Fig. 18. Fig. 22 shows a modified example of the functional blocks of the router 412 shown in Fig. 19. In the above modification C, the transmission prohibition control units 126, 412f may transmit a control signal ctl3 generated based on the transmission prohibition data 127, 412g to at least one of the vertical scanning circuit 102 and the horizontal scanning circuit 105, for example, as shown in Figs. 21 and 22.
 垂直走査回路102は、伝送禁止制御部126,412fから入力された制御信号ctl3に基づいて、画素アレイ回路101におけるラインごとの選択(データ出力)の可否を決定してもよい。垂直走査回路102は、選択(データ出力)が禁止されたラインを除く各ラインを選択してもよい。 The vertical scanning circuit 102 may determine whether or not to select (output data) each line in the pixel array circuit 101 based on the control signal ctl3 input from the transmission prohibition control unit 126, 412f. The vertical scanning circuit 102 may select each line except for the lines for which selection (output data) is prohibited.
 水平走査回路105は、伝送禁止制御部126,412fから入力された制御信号ctl3に基づいて、画素アレイ回路101から得られた一ライン分の各画素データDpの出力の可否を決定してもよい。水平走査回路105は、データ出力が禁止された画素データDpを除く各画素データDpを出力してもよい。 The horizontal scanning circuit 105 may determine whether or not to output each line of pixel data Dp obtained from the pixel array circuit 101 based on the control signal ctl3 input from the transmission prohibition control units 126 and 412f. The horizontal scanning circuit 105 may output each pixel data Dp except for pixel data Dp for which data output is prohibited.
 図23(A),図23(B),図23(C),図23(D)は、制御信号ctl3に基づいて決定されたデータ出力の可否を模式的に表したものである。画素アレイ部110A,310Aは、制御信号ctl3に基づいて、画素アレイ回路101における各受光画素Pからのデータ出力の可否をラインごとに決定してもよい。このとき、画素アレイ部110A,310Aは、例えば、図23(A)に示したように、選択(データ出力)が禁止されたラインを除く各ラインの画素データDpを出力する。画素アレイ部110A,310Aは、制御信号ctl3に基づいて、画素アレイ回路101からのデータ出力の可否を受光画素Pごとに決定してもよい。このとき、画素アレイ部110A,310Aは、例えば、図23(B),図23(C),図23(D)に示したように、データ出力が禁止された画素データDpを除く各画素データDpを出力してもよい。 23(A), 23(B), 23(C), and 23(D) are schematic diagrams showing whether data output is possible based on the control signal ctl3. The pixel array unit 110A, 310A may determine whether data output is possible from each light-receiving pixel P in the pixel array circuit 101 for each line based on the control signal ctl3. At this time, the pixel array unit 110A, 310A outputs pixel data Dp for each line except for lines for which selection (data output) is prohibited, for example, as shown in FIG. 23(A). The pixel array unit 110A, 310A may determine whether data output is possible from the pixel array circuit 101 for each light-receiving pixel P based on the control signal ctl3. At this time, the pixel array units 110A and 310A may output each pixel data Dp except for the pixel data Dp for which data output is prohibited, as shown in, for example, Figures 23(B), 23(C), and 23(D).
 このように、本変形例では、伝送禁止制御部126,412fから入力された制御信号ctl3に基づいて、画素アレイ部110A,310Aからのデータ出力の可否が決定される。これにより、不要なデータの出力をなくすことができるので、画素アレイ部110A,310Aから各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this manner, in this modified example, whether or not data is to be output from the pixel array units 110A, 310A is determined based on the control signal ctl3 input from the transmission prohibition control units 126, 412f. This makes it possible to eliminate the output of unnecessary data, thereby reducing congestion in data transmission from the pixel array units 110A, 310A to each processor 211. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
[変形例E]
 図24は、図6のセルCの機能ブロックの一変形例を表したものである。図25は、図12のセルCの機能ブロックの一変形例を表したものである。上記各実施の形態および上記変形例A,Bにおいて、プロセッサ211は、例えば、図24,図25に示したように、カウンタ211f、伝送禁止制御部211gおよび伝送禁止データ211hを更に有してもよい。
[Modification E]
Fig. 24 shows a modified example of the functional blocks of cell C in Fig. 6. Fig. 25 shows a modified example of the functional blocks of cell C in Fig. 12. In each of the above embodiments and modifications A and B, the processor 211 may further include a counter 211f, a transmission prohibition control unit 211g, and transmission prohibition data 211h, for example, as shown in Figs. 24 and 25.
 カウンタ211fは、ニューロンI/O部211aから入力されるスパイク信号の数を、ニューロン送信先アドレスごとにカウントする。伝送禁止制御部211gは、カウンタ211fにおいて、カウント数が所定の閾値を超えたとき、所定の閾値を超えたカウント数に対応するニューロン送信先アドレスを伝送禁止データ211hに書き込む。伝送禁止データ211hは、例えば、図20(A)に示したように、アドレスごとに伝送可を示す「1」か、伝送禁止を示す「0」が対応付けられたデータとなっている。 The counter 211f counts the number of spike signals input from the neuron I/O unit 211a for each neuron destination address. When the count number in the counter 211f exceeds a predetermined threshold, the transmission prohibition control unit 211g writes the neuron destination address corresponding to the count number that exceeds the predetermined threshold into the transmission prohibition data 211h. For example, as shown in FIG. 20(A), the transmission prohibition data 211h is data in which either "1" indicating transmission permitted or "0" indicating transmission prohibited is associated with each address.
 ルータ212,412は、伝送禁止データ211hに基づいて、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpの出力の可否を判定する。ルータ212,412は、例えば、ルーティングテーブル(TBL)に基づいて取得したアドレスと、伝送禁止データ211hとを対比する。 The router 212, 412 determines whether or not to output pixel data Dp obtained from the router 120, 120A or pixel array unit 310A based on the transmission prohibition data 211h. The router 212, 412 compares, for example, an address obtained based on a routing table (TBL) with the transmission prohibition data 211h.
 ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ211hにおいて伝送禁止に該当する場合、ルータ212,412は、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpの出力を禁止する。一方、ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ211hにおいて伝送可に該当する場合、ルータ212,412は、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpをプロセッサ211に出力する。 If the address obtained based on the routing table (TBL) corresponds to "transmission prohibited" in the transmission prohibition data 211h, the router 212, 412 prohibits the output of pixel data Dp obtained from the router 120, 120A or pixel array unit 310A. On the other hand, if the address obtained based on the routing table (TBL) corresponds to "transmission permitted" in the transmission prohibition data 211h, the router 212, 412 outputs the pixel data Dp obtained from the router 120, 120A or pixel array unit 310A to the processor 211.
 このように、本変形例では、伝送禁止制御部211gからの伝送禁止データ211hに基づいて、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpの出力の可否が判定される。これにより、不要なデータの出力をなくすことができるので、ルータ212,412からプロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this manner, in this modified example, whether or not to output pixel data Dp obtained from routers 120, 120A or pixel array unit 310A is determined based on transmission prohibition data 211h from transmission prohibition control unit 211g. This makes it possible to eliminate the output of unnecessary data, thereby reducing congestion in data transmission from routers 212, 412 to processor 211. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using SNN.
 本変形例において、伝送禁止制御部211gは、所定の時間が経過した後、伝送禁止データ211hをリセットしてもよい。 In this modified example, the transmission prohibition control unit 211g may reset the transmission prohibition data 211h after a predetermined time has elapsed.
 また、本変形例において、ルーティングテーブル(TBL)に基づいて取得したアドレスが伝送禁止データ211hにおいて伝送禁止に該当する場合、行先アドレス決定部212c,412cは、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpを、伝送禁止データ211hにおいて伝送可に該当するアドレスのニューロンに迂回してもよい。この迂回先のニューロンは、伝送禁止となっているニューロンに対応するアテンションデータを補助的に表現する機能を有する。プロセッサ211は、伝送禁止となっているニューロンが示すデータと、この迂回先のニューロンが示すデータ(アテンションデータ)とを対応付けて出力してもよい。このようにした場合には、新たな情報処理を提供することが可能となる。 Furthermore, in this modified example, if an address obtained based on the routing table (TBL) corresponds to transmission prohibited in the transmission prohibited data 211h, the destination address determination unit 212c, 412c may divert the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A to a neuron at an address corresponding to transmission permitted in the transmission prohibited data 211h. This diversion destination neuron has a function of auxiliary representing attention data corresponding to the neuron for which transmission is prohibited. The processor 211 may output the data indicated by the neuron for which transmission is prohibited and the data indicated by the diversion destination neuron (attention data) in association with each other. In this case, it becomes possible to provide new information processing.
 また、本変形例において、プロセッサ211は、カウンタ211fに格納されるカウント値の時間差分値(所定の周期ごとのカウント値の差)に基づいて、重み格納メモリ部211cの重み値を変更してもよい。このようにした場合には、カウント値の増加の度合いを調整することができるので、迂回の頻度を調整することができる。その結果、ルータ212,412からプロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 Furthermore, in this modified example, the processor 211 may change the weight value in the weight storage memory unit 211c based on the time difference value (the difference between the count values for each predetermined period) of the count values stored in the counter 211f. In this case, the degree of increase in the count value can be adjusted, and therefore the frequency of detours can be adjusted. As a result, congestion in data transmission from the routers 212 and 412 to the processor 211 can be reduced. Therefore, a further improvement in the processing speed can be achieved in signal processing using an SNN.
[変形例F]
 図26は、図6,図12,図24,図25のセルC内の複数のニューロンの動作状態の一例を表したものである。図26には、膜電位メモリ部211e内の各ニューロンの動作状態が例示される。膜電位メモリ部211e内の複数のニューロンのうち、一部のニューロン(以下、「主たるニューロン」)が、画素アレイ部110A,310A内の複数の受光画素Pに対応する。また、膜電位メモリ部211e内の複数のニューロンのうち、主たるニューロン以外の複数のニューロンが、主たるニューロンの迂回先のニューロンに対応する。図26において、動作状態が伝送禁止となっている複数のニューロンと、動作状態が伝送可となっている複数のニューロンとが例示される。動作状態が伝送禁止となっている複数のニューロンが、主たるニューロンに対応する。動作状態が伝送可となっている複数のニューロンが、迂回先のニューロンに対応する。
[Variation F]
FIG. 26 shows an example of the operation state of a plurality of neurons in the cell C of FIG. 6, FIG. 12, FIG. 24, and FIG. 25. FIG. 26 illustrates the operation state of each neuron in the membrane potential memory unit 211e. Among the plurality of neurons in the membrane potential memory unit 211e, some neurons (hereinafter, "main neurons") correspond to the plurality of light receiving pixels P in the pixel array units 110A and 310A. Among the plurality of neurons in the membrane potential memory unit 211e, the plurality of neurons other than the main neurons correspond to the detouring destination neurons of the main neurons. In FIG. 26, a plurality of neurons whose operation state is transmission prohibited and a plurality of neurons whose operation state is transmission enabled are illustrated. The plurality of neurons whose operation state is transmission prohibited correspond to the main neurons. The plurality of neurons whose operation state is transmission enabled correspond to the detouring destination neurons.
 上記各実施の形態およびそれらの変形例において、行先アドレス決定部212c,412cは、主たるニューロンに対応する各ニューロンの動作状態が伝送禁止となっている場合、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpを、迂回先の複数のニューロンに迂回してもよい。 In each of the above embodiments and their variations, the destination address determination unit 212c, 412c may route pixel data Dp obtained from the router 120, 120A or pixel array unit 310A to multiple destination neurons when the operating state of each neuron corresponding to the main neuron is prohibited from transmission.
 図27(A)、図27(B)、図27(C)は、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpの迂回方法の一例を表したものである。 Figures 27(A), 27(B), and 27(C) show an example of a method for bypassing pixel data Dp obtained from router 120, 120A or pixel array unit 310A.
 ルータ212,412は、ルーティングテーブル(TBL)に基づいて取得した伝送先のアドレスが伝送禁止に該当する場合、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpの伝送先(転送先)を、転送可に該当するアドレスのニューロン(迂回先のニューロン)に変更する。伝送先の複数のニューロンが伝送先の処理パイプラインに相当する。迂回先の複数のニューロンが迂回先の処理パイプラインに相当する。ルータ212,412は、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpを、伝送先および迂回先のアドレスと対応付けた送信データDAをプロセッサ211に出力する。 If the destination address obtained based on the routing table (TBL) corresponds to a transmission prohibited address, the router 212, 412 changes the destination (transfer destination) of the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A to a neuron (a detouring neuron) whose address corresponds to a transmission permitted address. The multiple neurons at the destination correspond to a processing pipeline at the destination. The multiple neurons at the detouring destination correspond to a processing pipeline at the detouring destination. The router 212, 412 outputs transmission data DA to the processor 211, in which the pixel data Dp obtained from the router 120, 120A or the pixel array unit 310A corresponds to the destination and detouring addresses.
 ニューロンI/O部211aは、ルータ212,412から受け取った送信データDAに含まれる画素データDpをスパイク信号として積和演算部211bに出力する。ニューロンI/O部211aは、ルータ212から受け取った画素データDpを、ルータ212から受け取った迂回先のアドレスと対応付けて積和演算部211bに出力する。積和演算部211bは、ニューロンI/O部211aから入力されるスパイク信号に対して、伝送先のアドレスごとに設定された所定の重み値を掛け、迂回先のアドレスごとに入力スパイク数を加算する積和演算を行う。積和演算部211bは、そのようにして得られた演算結果を、ニューロンI/O部211aを介して膜電位メモリ部211dに格納する。上記の重み値は、重み格納メモリ部211cに格納される。ニューロンI/O部211aは、積和演算部211bから受け取った、迂回先のアドレスごとの値(積和演算の結果)を膜電位として膜電位メモリ部211dに格納する。 The neuron I/O unit 211a outputs pixel data Dp contained in the transmission data DA received from the routers 212 and 412 as a spike signal to the product-sum calculation unit 211b. The neuron I/O unit 211a outputs the pixel data Dp received from the router 212 to the product-sum calculation unit 211b in association with the address of the detouring destination received from the router 212. The product-sum calculation unit 211b multiplies the spike signal input from the neuron I/O unit 211a by a predetermined weight value set for each destination address, and performs a product-sum calculation to add the number of input spikes for each detouring destination address. The product-sum calculation unit 211b stores the calculation results thus obtained in the membrane potential memory unit 211d via the neuron I/O unit 211a. The weight value is stored in the weight storage memory unit 211c. The neuron I/O unit 211a stores the value (result of the product-sum calculation) for each address of the detour destination received from the product-sum calculation unit 211b as a membrane potential in the membrane potential memory unit 211d.
 ルータ212,412は、プロセッサ211内のLIF部211dから得られたデータ(スパイク信号およびアドレス)に基づいて、当該ルータ212,412に対応するプロセッサ211内のニューロンの動作状態についてのデータを生成する。ルータ212,412は、生成したプロセッサ211内のニューロンの動作状態についてのデータに基づいて、伝送禁止データ127,412gを生成(更新)する。 The routers 212, 412 generate data on the operating state of the neurons in the processor 211 corresponding to the router 212, 412 based on the data (spike signals and addresses) obtained from the LIF unit 211d in the processor 211. The routers 212, 412 generate (update) transmission prohibition data 127, 412g based on the generated data on the operating state of the neurons in the processor 211.
 ルータ212,412は、生成(更新)した伝送禁止データ127,412gに基づいて、伝送禁止に該当するアドレスのステータス(伝送禁止または伝送可)を判定する。ルータ212,412は、例えば、伝送禁止に該当するアドレスと、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gとを対比する。伝送禁止に該当するアドレスが、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gにおいて伝送禁止に該当する場合、ルータ212,412は、例えば、伝送禁止に該当するアドレスのステータスを引き続き、伝送禁止とする。一方、伝送禁止に該当するアドレスが、伝送禁止制御部126,412fから得られた伝送禁止データ127,412gにおいて伝送可に該当する場合、ルータ212,412は、例えば、伝送禁止に該当するアドレスのステータスを伝送可に変更する。 The router 212, 412 determines the status (transmission prohibited or transmission possible) of the address corresponding to the transmission prohibition based on the generated (updated) transmission prohibition data 127, 412g. The router 212, 412, for example, compares the address corresponding to the transmission prohibition with the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f. If the address corresponding to the transmission prohibition corresponds to the transmission prohibition in the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f, the router 212, 412, for example, continues to set the status of the address corresponding to the transmission prohibition to transmission prohibited. On the other hand, if the address corresponding to the transmission prohibition corresponds to the transmission possible in the transmission prohibition data 127, 412g obtained from the transmission prohibition control unit 126, 412f, the router 212, 412, for example, changes the status of the address corresponding to the transmission prohibition to transmission possible.
 伝送禁止に該当するアドレスのステータスが伝送可に変更された場合、ルータ212,412は、例えば、図27(A)に示したように、迂回先の処理パイプラインでの処理結果を伝送先の各処理パイプラインに送信してもよい。伝送禁止に該当するアドレスのステータスが伝送可に変更された場合、ルータ212,412は、例えば、図27(B)に示したように、迂回先の処理パイプラインでの処理結果を伝送先の1つの処理パイプラインに送信してもよい。このとき、ルータ212,412は、例えば、図27(C)に示したように、迂回先での処理結果についてのデータを制御信号ctlとして出力してもよい。 When the status of an address corresponding to transmission prohibition is changed to transmission possible, the router 212, 412 may transmit the processing result of the processing pipeline of the detouring destination to each processing pipeline of the transmission destination, for example, as shown in FIG. 27(A). When the status of an address corresponding to transmission prohibition is changed to transmission possible, the router 212, 412 may transmit the processing result of the processing pipeline of the detouring destination to one processing pipeline of the transmission destination, for example, as shown in FIG. 27(B). At this time, the router 212, 412 may output data on the processing result at the detouring destination as a control signal ctl, for example, as shown in FIG. 27(C).
 本変形例では、主たるニューロンに対応する各ニューロンの動作状態が伝送禁止となっている場合、ルータ120,120Aまたは画素アレイ部310Aから得られた画素データDpが迂回先の複数のニューロンに迂回される。これにより、画素アレイ部310から各プロセッサ211へのデータ送信における輻輳を低減することができる。従って、SNNを用いた信号処理において、更なる処理速度の向上を実現することができる。 In this modified example, when the operating state of each neuron corresponding to the main neuron is transmission prohibited, pixel data Dp obtained from routers 120, 120A or pixel array unit 310A is diverted to multiple neurons at the diverting destination. This makes it possible to reduce congestion in data transmission from pixel array unit 310 to each processor 211. Therefore, it is possible to achieve a further improvement in processing speed in signal processing using an SNN.
[変形例G]
 図28は、図1の撮像装置1000の一変形例を表したものである。上記第1の実施の形態およびその変形例において、画素アレイ部110は、例えば、図28に示したように、二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成されてもよい。複数の受光画素P1および複数の受光画素P2は、例えば、行方向および列方向に交互に配置される。
[Modification G]
Fig. 28 shows a modified example of the imaging device 1000 of Fig. 1. In the above-described first embodiment and its modified examples, the pixel array section 110 may be configured to include a plurality of light receiving pixels P1 arranged two-dimensionally and a plurality of light receiving pixels P2 arranged two-dimensionally, for example, as shown in Fig. 28. The plurality of light receiving pixels P1 and the plurality of light receiving pixels P2 are arranged alternately in the row direction and the column direction, for example.
 受光画素P1は、例えば、CMOS素子もしくはSPAD素子を含んでもよい。このとき、複数の受光画素P1からなる第1の画素アレイは、外部から入射する可視波長帯の光を複数の受光画素P1で検出することにより複数のデジタル信号(画素データDp1)をルータ120に出力する。第1の画素アレイは、所定の周期Taで複数のデジタル信号(画素データDp1)をルータ120に出力する。ルータ120は、画素アレイ部110から複数の画素データDp1を取得すると、取得した複数の画素データDp1を含む送信データDAを各コアCへ送信するとともに、取得した複数の画素データDp1をデジタルの可視光画像データIout1(例えばRGB画像データ)としてエンコーダ510に出力する。 The light receiving pixel P1 may include, for example, a CMOS element or a SPAD element. At this time, the first pixel array consisting of a plurality of light receiving pixels P1 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P1, and outputs a plurality of digital signals (pixel data Dp1) to the router 120. The first pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120 at a predetermined period Ta. When the router 120 acquires a plurality of pixel data Dp1 from the pixel array unit 110, it transmits transmission data DA including the acquired plurality of pixel data Dp1 to each core C, and outputs the acquired plurality of pixel data Dp1 to the encoder 510 as digital visible light image data Iout1 (for example, RGB image data).
 受光画素P2は、例えば、EVS素子を含んでもよい。このとき、複数の受光画素P2からなる第2の画素アレイは、外部から入射する可視波長帯の光を複数の受光画素P2で検出することにより複数のデジタル信号(画素データDp2)を出力する。第2の画素アレイは、所定の周期Tb(<Ta)で複数のデジタル信号(画素データDp1)をルータ120に出力する。ルータ120は、画素アレイ部110から複数の画素データDp2を取得すると、取得した複数の画素データDp2を含む送信データDAを各コアCへ送信するとともに、取得した複数の受光画素P2をデジタルのEVS画像データIout2としてエンコーダ510に出力する。 The light receiving pixel P2 may include, for example, an EVS element. At this time, the second pixel array consisting of a plurality of light receiving pixels P2 detects light in the visible wavelength band incident from the outside with the plurality of light receiving pixels P2, and outputs a plurality of digital signals (pixel data Dp2). The second pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120 at a predetermined period Tb (<Ta). When the router 120 acquires the plurality of pixel data Dp2 from the pixel array section 110, it transmits transmission data DA including the acquired plurality of pixel data Dp2 to each core C, and outputs the acquired plurality of light receiving pixels P2 to the encoder 510 as digital EVS image data Iout2.
 画素アレイ部110は、例えば、図29に示したように、可視光画像データIout1(例えばRGB画像データ)をルータ120に出力した後、1または複数のEVS画像データIout2をルータ120に出力する。画素アレイ部110は、例えば、可視光画像データIout1(例えばRGB画像データ)および1または複数のEVS画像データIout2を周期Taでルータ120に出力する。エンコーダ510は、入力された可視光画像データIout1(例えばRGB画像データ)をエンコードし、それにより得られた特徴量画像データC1を送信部520に出力する。エンコーダ510は、入力されたEVS画像データIout2をエンコードし、それにより得られた特徴量画像データC2を送信部520に出力する。 29, the pixel array unit 110 outputs visible light image data Iout1 (e.g., RGB image data) to the router 120, and then outputs one or more EVS image data Iout2 to the router 120. The pixel array unit 110 outputs, for example, visible light image data Iout1 (e.g., RGB image data) and one or more EVS image data Iout2 to the router 120 at a period Ta. The encoder 510 encodes the input visible light image data Iout1 (e.g., RGB image data) and outputs the resulting feature image data C1 to the transmitter 520. The encoder 510 encodes the input EVS image data Iout2 and outputs the resulting feature image data C2 to the transmitter 520.
 情報処理装置3000は、送信部520と通信することの可能な受信部3100と、受信部3100で取得した特徴量画像データC1および特徴量画像データC2をデコードするデコーダ3200とを備える。デコーダ3200は、特徴量画像データC1をデコードすることにより復元可視光画像データIout1’(例えばRGB画像データ)を生成する。デコーダ3200は、特徴量画像データC2をデコードすることにより復元EVS画像データIout2’を生成する。 The information processing device 3000 includes a receiving unit 3100 capable of communicating with the transmitting unit 520, and a decoder 3200 that decodes the feature amount image data C1 and feature amount image data C2 acquired by the receiving unit 3100. The decoder 3200 generates restored visible light image data Iout1' (e.g., RGB image data) by decoding the feature amount image data C1. The decoder 3200 generates restored EVS image data Iout2' by decoding the feature amount image data C2.
 情報処理装置3000は、さらに、デコーダ3200で生成されたデータを処理するデータ処理部3300を備える。データ処理部3300は、例えば、図29に示したように、可復元可視光画像データIout1’と、復元EVS画像データIout2’とに基づいて、補完可視光画像データIout2’’を生成する。補完可視光画像データIout2’’は、周期的に生成される複数の復元可視光画像データIout1’を補完するデータである。 The information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3200. For example, as shown in FIG. 29, the data processing unit 3300 generates complementary visible light image data Iout2'' based on the restorable visible light image data Iout1' and the restored EVS image data Iout2'. The complementary visible light image data Iout2'' is data that complements the multiple restored visible light image data Iout1' that are generated periodically.
 本変形例では、画素アレイ部110が二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成される。これにより、複数の受光画素P1から得られた可視光画像データIout1、および複数の受光画素P2から得られた1または複数のEVS画像データIout2を周期的にルータ120に出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。 In this modified example, the pixel array section 110 is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout1 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2, to the router 120. As a result, it is possible to reduce the amount of data transmitted from the imaging device 1000 to the information processing device 3000.
[変形例H]
 図30は、図1の撮像装置1000の一変形例を表したものである。上記変形例Gにおいて、画素アレイ部110は、例えば、図30に示したように、エンコーダ510および送信部520の代わりに、エンコーダ530および通信部540が設けられてもよい。このとき、エンコーダ530は、可視光画像データIout1に基づいてコアアレイ部210で得られたデータDout3をエンコードし、それにより得られた特徴量画像データC3を通信部540に出力する。エンコーダ530は、EVS画像データIout2に基づいてコアアレイ部210で得られたデータDout4をエンコードし、それにより得られた特徴量画像データC4を通信部540に出力する。
[Variation H]
Fig. 30 shows a modified example of the imaging device 1000 of Fig. 1. In the modified example G, the pixel array unit 110 may be provided with an encoder 530 and a communication unit 540 instead of the encoder 510 and the transmission unit 520, as shown in Fig. 30. In this case, the encoder 530 encodes the data Dout3 obtained by the core array unit 210 based on the visible light image data Iout1, and outputs the resulting feature image data C3 to the communication unit 540. The encoder 530 encodes the data Dout4 obtained by the core array unit 210 based on the EVS image data Iout2, and outputs the resulting feature image data C4 to the communication unit 540.
 情報処理装置3000は、通信部540と通信することの可能な受信部3400と、受信部3400で取得した特徴量画像データC3および特徴量画像データC4をデコードするデコーダ3500とを備える。デコーダ3500は、特徴量画像データC3をデコードすることにより復元可視光画像データIout3’(例えばRGB画像データ)を生成する。デコーダ3500は、特徴量画像データC4をデコードすることにより復元EVS画像データIout4’を生成する。 The information processing device 3000 includes a receiving unit 3400 capable of communicating with the communication unit 540, and a decoder 3500 that decodes the feature amount image data C3 and feature amount image data C4 acquired by the receiving unit 3400. The decoder 3500 generates restored visible light image data Iout3' (e.g., RGB image data) by decoding the feature amount image data C3. The decoder 3500 generates restored EVS image data Iout4' by decoding the feature amount image data C4.
 情報処理装置3000は、さらに、デコーダ3500で生成されたデータを処理するデータ処理部3300を備える。データ処理部3300は、例えば、可復元可視光画像データIout3’と、復元EVS画像データIout4’とに基づいて、補完可視光画像データIout4’’を生成する。補完可視光画像データIout4’’は、周期的に生成される複数の復元可視光画像データIout4’を補完するデータである。 The information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3500. The data processing unit 3300 generates complementary visible light image data Iout4'' based on, for example, the restorable visible light image data Iout3' and the restored EVS image data Iout4'. The complementary visible light image data Iout4'' is data that complements multiple restored visible light image data Iout4' that are generated periodically.
 本変形例では、画素アレイ部110が二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成される。これにより、複数の受光画素P1から得られた可視光画像データIout3、および複数の受光画素P2から得られた1または複数のEVS画像データIout4を周期的にルータ120に出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。 In this modified example, the pixel array section 110 is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout3 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2, to the router 120. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
[変形例I]
 図31は、図1の撮像装置1000の一変形例を表したものである。上記変形例Gにおいて、画素アレイ部110は、例えば、図31に示したように、エンコーダ530および通信部540を更に備えてもよい。このようにした場合には、複数の受光画素P1から得られた可視光画像データIout1、および複数の受光画素P2から得られた1または複数のEVS画像データIout2を周期的にルータ120に出力することができる。さらに、複数の受光画素P1から得られた可視光画像データIout3、および複数の受光画素P2から得られた1または複数のEVS画像データIout4を周期的にルータ120に出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。
[Variation I]
31 shows a modified example of the imaging device 1000 of FIG. 1. In the modified example G, the pixel array section 110 may further include an encoder 530 and a communication section 540, for example, as shown in FIG. 31. In this case, the visible light image data Iout1 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120. Furthermore, the visible light image data Iout3 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120. As a result, the amount of data transmission from the imaging device 1000 to the information processing device 3000 can be reduced.
[変形例J]
 図32は、図7の撮像装置1000の一変形例を表したものである。上記第1の実施の形態の変形例において、画素アレイ部110Aは、例えば、図32に示したように、二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成されてもよい。複数の受光画素P1および複数の受光画素P2は、例えば、行方向および列方向に交互に配置される。
[Modification J]
Fig. 32 shows a modified example of the imaging device 1000 of Fig. 7. In the modified example of the first embodiment, the pixel array section 110A may be configured to include a plurality of light receiving pixels P1 arranged two-dimensionally and a plurality of light receiving pixels P2 arranged two-dimensionally, for example, as shown in Fig. 32. The plurality of light receiving pixels P1 and the plurality of light receiving pixels P2 are arranged alternately in the row direction and the column direction, for example.
 受光画素P1は、例えば、CMOS素子もしくはSPAD素子を含んでもよい。このとき、複数の受光画素P1からなる第1の画素アレイは、外部から入射する可視波長帯の光を複数の受光画素P1で検出することにより複数のデジタル信号(画素データDp1)をルータ120Aに出力する。第1の画素アレイは、所定の周期Taで複数のデジタル信号(画素データDp1)をルータ120Aに出力する。ルータ120Aは、画素アレイ部110Aから複数の画素データDp1を取得すると、取得した複数の画素データDp1を含む送信データDAをルータ210Aへ送信するとともに、取得した複数の画素データDp1をデジタルの可視光画像データIout1(例えばRGB画像データ)としてエンコーダ510に出力する。 The light receiving pixel P1 may include, for example, a CMOS element or a SPAD element. At this time, the first pixel array consisting of a plurality of light receiving pixels P1 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P1, and outputs a plurality of digital signals (pixel data Dp1) to the router 120A. The first pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120A at a predetermined period Ta. When the router 120A acquires a plurality of pixel data Dp1 from the pixel array section 110A, it transmits transmission data DA including the acquired plurality of pixel data Dp1 to the router 210A, and outputs the acquired plurality of pixel data Dp1 to the encoder 510 as digital visible light image data Iout1 (for example, RGB image data).
 受光画素P2は、例えば、EVS素子を含んでもよい。このとき、複数の受光画素P2からなる第2の画素アレイは、外部から入射する可視波長帯の光を複数の受光画素P2で検出することにより複数のデジタル信号(画素データDp2)を出力する。第2の画素アレイは、所定の周期Tb(<Ta)で複数のデジタル信号(画素データDp1)をルータ120Aに出力する。ルータ120Aは、画素アレイ部110Aから複数の画素データDp2を取得すると、取得した複数の画素データDp2を含む送信データDAをルータ210Aへ送信するとともに、取得した複数の受光画素P2をデジタルのEVS画像データIout2としてエンコーダ510に出力する。 The light receiving pixel P2 may include, for example, an EVS element. At this time, the second pixel array consisting of a plurality of light receiving pixels P2 detects light in the visible wavelength band incident from the outside at the plurality of light receiving pixels P2 and outputs a plurality of digital signals (pixel data Dp2). The second pixel array outputs a plurality of digital signals (pixel data Dp1) to the router 120A at a predetermined period Tb (<Ta). When the router 120A acquires the plurality of pixel data Dp2 from the pixel array section 110A, it transmits transmission data DA including the acquired plurality of pixel data Dp2 to the router 210A and outputs the acquired plurality of light receiving pixels P2 to the encoder 510 as digital EVS image data Iout2.
 画素アレイ部110Aは、例えば、図32に示したように、可視光画像データIout1(例えばRGB画像データ)をルータ120Aに出力した後、1または複数のEVS画像データIout2をルータ120Aに出力する。画素アレイ部110Aは、例えば、可視光画像データIout1(例えばRGB画像データ)および1または複数のEVS画像データIout2を周期Taでルータ120Aに出力する。エンコーダ510は、入力された可視光画像データIout1(例えばRGB画像データ)をエンコードし、それにより得られた特徴量画像データC1を送信部520に出力する。エンコーダ510は、入力されたEVS画像データIout2をエンコードし、それにより得られた特徴量画像データC2を送信部520に出力する。 For example, as shown in FIG. 32, the pixel array unit 110A outputs visible light image data Iout1 (e.g., RGB image data) to the router 120A, and then outputs one or more EVS image data Iout2 to the router 120A. The pixel array unit 110A outputs, for example, visible light image data Iout1 (e.g., RGB image data) and one or more EVS image data Iout2 to the router 120A at a period Ta. The encoder 510 encodes the input visible light image data Iout1 (e.g., RGB image data) and outputs the resulting feature image data C1 to the transmitter 520. The encoder 510 encodes the input EVS image data Iout2 and outputs the resulting feature image data C2 to the transmitter 520.
 情報処理装置3000は、送信部520と通信することの可能な受信部3100と、受信部3100で取得した特徴量画像データC1および特徴量画像データC2をデコードするデコーダ3200とを備える。デコーダ3200は、特徴量画像データC1をデコードすることにより復元可視光画像データIout1’(例えばRGB画像データ)を生成する。デコーダ3200は、特徴量画像データC2をデコードすることにより復元EVS画像データIout2’を生成する。 The information processing device 3000 includes a receiving unit 3100 capable of communicating with the transmitting unit 520, and a decoder 3200 that decodes the feature amount image data C1 and feature amount image data C2 acquired by the receiving unit 3100. The decoder 3200 generates restored visible light image data Iout1' (e.g., RGB image data) by decoding the feature amount image data C1. The decoder 3200 generates restored EVS image data Iout2' by decoding the feature amount image data C2.
 情報処理装置3000は、さらに、デコーダ3200で生成されたデータを処理するデータ処理部3300を備える。データ処理部3300は、例えば、図29に示したように、可復元可視光画像データIout1’と、復元EVS画像データIout2’とに基づいて、補完可視光画像データIout2’’を生成する。補完可視光画像データIout2’’は、周期的に生成される複数の復元可視光画像データIout1’を補完するデータである。 The information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3200. For example, as shown in FIG. 29, the data processing unit 3300 generates complementary visible light image data Iout2'' based on the restorable visible light image data Iout1' and the restored EVS image data Iout2'. The complementary visible light image data Iout2'' is data that complements the multiple restored visible light image data Iout1' that are generated periodically.
 本変形例では、画素アレイ部110Aが二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成される。これにより、複数の受光画素P1から得られた可視光画像データIout1、および複数の受光画素P2から得られた1または複数のEVS画像データIout2を周期的にルータ120Aに出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。 In this modified example, the pixel array section 110A is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout1 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2, to the router 120A. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
[変形例K]
 図33は、図1の撮像装置1000の一変形例を表したものである。上記変形例Jにおいて、画素アレイ部110Aは、例えば、図33に示したように、エンコーダ510および送信部520の代わりに、エンコーダ530および通信部540が設けられてもよい。このとき、エンコーダ530は、可視光画像データIout1に基づいてコアアレイ部210Aで得られたデータDout3をエンコードし、それにより得られた特徴量画像データC3を通信部540に出力する。エンコーダ530は、EVS画像データIout2に基づいてコアアレイ部210Aで得られたデータDout4をエンコードし、それにより得られた特徴量画像データC4を通信部540に出力する。
[Modification K]
Fig. 33 shows a modified example of the imaging device 1000 of Fig. 1. In the above modified example J, the pixel array unit 110A may be provided with an encoder 530 and a communication unit 540 instead of the encoder 510 and the transmission unit 520, as shown in Fig. 33. In this case, the encoder 530 encodes the data Dout3 obtained by the core array unit 210A based on the visible light image data Iout1, and outputs the resulting feature image data C3 to the communication unit 540. The encoder 530 encodes the data Dout4 obtained by the core array unit 210A based on the EVS image data Iout2, and outputs the resulting feature image data C4 to the communication unit 540.
 情報処理装置3000は、通信部540と通信することの可能な受信部3400と、受信部3400で取得した特徴量画像データC3および特徴量画像データC4をデコードするデコーダ3500とを備える。デコーダ3500は、特徴量画像データC3をデコードすることにより復元可視光画像データIout3’(例えばRGB画像データ)を生成する。デコーダ3500は、特徴量画像データC4をデコードすることにより復元EVS画像データIout4’を生成する。 The information processing device 3000 includes a receiving unit 3400 capable of communicating with the communication unit 540, and a decoder 3500 that decodes the feature amount image data C3 and feature amount image data C4 acquired by the receiving unit 3400. The decoder 3500 generates restored visible light image data Iout3' (e.g., RGB image data) by decoding the feature amount image data C3. The decoder 3500 generates restored EVS image data Iout4' by decoding the feature amount image data C4.
 情報処理装置3000は、さらに、デコーダ3500で生成されたデータを処理するデータ処理部3300を備える。データ処理部3300は、例えば、可復元可視光画像データIout3’と、復元EVS画像データIout4’とに基づいて、補完可視光画像データIout4’’を生成する。補完可視光画像データIout4’’は、周期的に生成される複数の復元可視光画像データIout4’を補完するデータである。 The information processing device 3000 further includes a data processing unit 3300 that processes the data generated by the decoder 3500. The data processing unit 3300 generates complementary visible light image data Iout4'' based on, for example, the restorable visible light image data Iout3' and the restored EVS image data Iout4'. The complementary visible light image data Iout4'' is data that complements multiple restored visible light image data Iout4' that are generated periodically.
 本変形例では、画素アレイ部110Aが二次元配置された複数の受光画素P1と、二次元配置された複数の受光画素P2とを含んで構成される。これにより、複数の受光画素P1から得られた可視光画像データIout3、および複数の受光画素P2から得られた1または複数のEVS画像データIout4を周期的にルータ120Aに出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。 In this modified example, the pixel array section 110A is configured to include a plurality of light receiving pixels P1 arranged two-dimensionally, and a plurality of light receiving pixels P2 arranged two-dimensionally. This makes it possible to periodically output visible light image data Iout3 obtained from the plurality of light receiving pixels P1, and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2, to the router 120A. As a result, the amount of data transmitted from the imaging device 1000 to the information processing device 3000 can be reduced.
[変形例L]
 図34は、図1の撮像装置1000の一変形例を表したものである。上記変形例Jにおいて、画素アレイ部110Aは、例えば、図34に示したように、エンコーダ530および通信部540を更に備えてもよい。このようにした場合には、複数の受光画素P1から得られた可視光画像データIout1、および複数の受光画素P2から得られた1または複数のEVS画像データIout2を周期的にルータ120Aに出力することができる。さらに、複数の受光画素P1から得られた可視光画像データIout3、および複数の受光画素P2から得られた1または複数のEVS画像データIout4を周期的にルータ120Aに出力することができる。その結果、撮像装置1000から情報処理装置3000へのデータ伝送量を低減することができる。
[Variation L]
FIG. 34 shows a modified example of the imaging device 1000 of FIG. 1. In the above modified example J, the pixel array section 110A may further include an encoder 530 and a communication section 540, for example, as shown in FIG. 34. In this case, the visible light image data Iout1 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout2 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120A. Furthermore, the visible light image data Iout3 obtained from the plurality of light receiving pixels P1 and one or more EVS image data Iout4 obtained from the plurality of light receiving pixels P2 can be periodically output to the router 120A. As a result, the amount of data transmission from the imaging device 1000 to the information processing device 3000 can be reduced.
<5.撮像装置の使用例>
 図35は、上記実施の形態およびその変形例に係る撮像装置1の使用例を表すものである。上述した撮像装置1000は、例えば、以下のように、可視光や、赤外光、紫外光、X線等の光をセンシングする様々なケースに使用することができる。
5. Examples of use of imaging device
35 shows an example of use of the imaging device 1 according to the above embodiment and its modified example. The imaging device 1000 described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as described below.
・ディジタルカメラや、カメラ機能付きの携帯機器等の、鑑賞の用に供される画像を撮影する装置
・自動停止等の安全運転や、運転者の状態の認識等のために、自動車の前方や後方、周囲、車内等を撮影する車載用センサ、走行車両や道路を監視する監視カメラ、車両間等の測距を行う測距センサ等の、交通の用に供される装置
・ユーザのジェスチャを撮影して、そのジェスチャに従った機器操作を行うために、テレビジョンや、冷蔵庫、エアーコンディショナ等の家電に供される装置
・内視鏡や、赤外光の受光による血管撮影を行う装置等の、医療やヘルスケアの用に供される装置
・防犯用途の監視カメラや、人物認証用途のカメラ等の、セキュリティの用に供される装置
・肌を撮影する肌測定器や、頭皮を撮影するマイクロスコープ等の、美容の用に供される装置
・スポーツ用途等向けのアクションカメラやウェアラブルカメラ等の、スポーツの用に供される装置
・畑や作物の状態を監視するためのカメラ等の、農業の用に供される装置
- Devices for taking images for viewing, such as digital cameras and mobile devices with camera functions. - Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. - Devices for home appliances such as televisions, refrigerators, and air conditioners that take images of users' gestures and operate the equipment according to those gestures. - Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. - Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. - Devices for sports purposes, such as action cameras and wearable cameras for sports purposes. - Devices for agriculture, such as cameras for monitoring the condition of fields and crops.
<6.移動体への応用例>
 本開示に係る技術(本技術)は、様々な製品へ応用することができる。例えば、本開示に係る技術は、自動車、電気自動車、ハイブリッド電気自動車、自動二輪車、自転車、パーソナルモビリティ、飛行機、ドローン、船舶、ロボット等のいずれかの種類の移動体に搭載される装置として実現されてもよい。
<6. Examples of applications to moving objects>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, or a robot.
 図36は、本開示に係る技術が適用され得る移動体制御システムの一例である車両制御システムの概略的な構成例を示すブロック図である。 FIG. 36 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology disclosed herein can be applied.
 車両制御システム12000は、通信ネットワーク12001を介して接続された複数の電子制御ユニットを備える。図36に示した例では、車両制御システム12000は、駆動系制御ユニット12010、ボディ系制御ユニット12020、車外情報検出ユニット12030、車内情報検出ユニット12040、及び統合制御ユニット12050を備える。また、統合制御ユニット12050の機能構成として、マイクロコンピュータ12051、音声画像出力部12052、及び車載ネットワークI/F(interface)12053が図示されている。 The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in FIG. 36, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside vehicle information detection unit 12030, an inside vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio/video output unit 12052, and an in-vehicle network I/F (interface) 12053.
 駆動系制御ユニット12010は、各種プログラムにしたがって車両の駆動系に関連する装置の動作を制御する。例えば、駆動系制御ユニット12010は、内燃機関又は駆動用モータ等の車両の駆動力を発生させるための駆動力発生装置、駆動力を車輪に伝達するための駆動力伝達機構、車両の舵角を調節するステアリング機構、及び、車両の制動力を発生させる制動装置等の制御装置として機能する。 The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 functions as a control device for a drive force generating device for generating the drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force for the vehicle.
 ボディ系制御ユニット12020は、各種プログラムにしたがって車体に装備された各種装置の動作を制御する。例えば、ボディ系制御ユニット12020は、キーレスエントリシステム、スマートキーシステム、パワーウィンドウ装置、あるいは、ヘッドランプ、バックランプ、ブレーキランプ、ウィンカー又はフォグランプ等の各種ランプの制御装置として機能する。この場合、ボディ系制御ユニット12020には、鍵を代替する携帯機から発信される電波又は各種スイッチの信号が入力され得る。ボディ系制御ユニット12020は、これらの電波又は信号の入力を受け付け、車両のドアロック装置、パワーウィンドウ装置、ランプ等を制御する。 The body system control unit 12020 controls the operation of various devices installed in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, tail lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals from various switches transmitted from a portable device that replaces a key can be input to the body system control unit 12020. The body system control unit 12020 accepts the input of these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.
 車外情報検出ユニット12030は、車両制御システム12000を搭載した車両の外部の情報を検出する。例えば、車外情報検出ユニット12030には、撮像部12031が接続される。車外情報検出ユニット12030は、撮像部12031に車外の画像を撮像させるとともに、撮像された画像を受信する。車外情報検出ユニット12030は、受信した画像に基づいて、人、車、障害物、標識又は路面上の文字等の物体検出処理又は距離検出処理を行ってもよい。 The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image capturing unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the image capturing unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, or characters on the road surface based on the received images.
 撮像部12031は、光を受光し、その光の受光量に応じた電気信号を出力する光センサである。撮像部12031は、電気信号を画像として出力することもできるし、測距の情報として出力することもできる。また、撮像部12031が受光する光は、可視光であっても良いし、赤外線等の非可視光であっても良い。 The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared light.
 車内情報検出ユニット12040は、車内の情報を検出する。車内情報検出ユニット12040には、例えば、運転者の状態を検出する運転者状態検出部12041が接続される。運転者状態検出部12041は、例えば運転者を撮像するカメラを含み、車内情報検出ユニット12040は、運転者状態検出部12041から入力される検出情報に基づいて、運転者の疲労度合い又は集中度合いを算出してもよいし、運転者が居眠りをしていないかを判別してもよい。 The in-vehicle information detection unit 12040 detects information inside the vehicle. To the in-vehicle information detection unit 12040, for example, a driver state detection unit 12041 that detects the state of the driver is connected. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the driver's degree of fatigue or concentration based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.
 マイクロコンピュータ12051は、車外情報検出ユニット12030又は車内情報検出ユニット12040で取得される車内外の情報に基づいて、駆動力発生装置、ステアリング機構又は制動装置の制御目標値を演算し、駆動系制御ユニット12010に対して制御指令を出力することができる。例えば、マイクロコンピュータ12051は、車両の衝突回避あるいは衝撃緩和、車間距離に基づく追従走行、車速維持走行、車両の衝突警告、又は車両のレーン逸脱警告等を含むADAS(Advanced Driver Assistance System)の機能実現を目的とした協調制御を行うことができる。 The microcomputer 12051 can calculate the control target values of the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.
 また、マイクロコンピュータ12051は、車外情報検出ユニット12030又は車内情報検出ユニット12040で取得される車両の周囲の情報に基づいて駆動力発生装置、ステアリング機構又は制動装置等を制御することにより、運転者の操作に拠らずに自律的に走行する自動運転等を目的とした協調制御を行うことができる。 The microcomputer 12051 can also control the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040, thereby performing cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on the driver's operation.
 また、マイクロコンピュータ12051は、車外情報検出ユニット12030で取得される車外の情報に基づいて、ボディ系制御ユニット12020に対して制御指令を出力することができる。例えば、マイクロコンピュータ12051は、車外情報検出ユニット12030で検知した先行車又は対向車の位置に応じてヘッドランプを制御し、ハイビームをロービームに切り替える等の防眩を図ることを目的とした協調制御を行うことができる。 The microcomputer 12051 can also output control commands to the body system control unit 12020 based on information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching high beams to low beams.
 音声画像出力部12052は、車両の搭乗者又は車外に対して、視覚的又は聴覚的に情報を通知することが可能な出力装置へ音声及び画像のうちの少なくとも一方の出力信号を送信する。図36の例では、出力装置として、オーディオスピーカ12061、表示部12062及びインストルメントパネル12063が例示されている。表示部12062は、例えば、オンボードディスプレイ及びヘッドアップディスプレイの少なくとも一つを含んでいてもよい。 The audio/image output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying the occupants of the vehicle or the outside of the vehicle of information. In the example of FIG. 36, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.
 図37は、撮像部12031の設置位置の例を示す図である。 FIG. 37 shows an example of the installation position of the imaging unit 12031.
 図37では、車両12100は、撮像部12031として、撮像部12101,12102,12103,12104,12105を有する。 In FIG. 37, the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.
 撮像部12101,12102,12103,12104,12105は、例えば、車両12100のフロントノーズ、サイドミラー、リアバンパ、バックドア及び車室内のフロントガラスの上部等の位置に設けられる。フロントノーズに備えられる撮像部12101及び車室内のフロントガラスの上部に備えられる撮像部12105は、主として車両12100の前方の画像を取得する。サイドミラーに備えられる撮像部12102,12103は、主として車両12100の側方の画像を取得する。リアバンパ又はバックドアに備えられる撮像部12104は、主として車両12100の後方の画像を取得する。撮像部12101及び12105で取得される前方の画像は、主として先行車両又は、歩行者、障害物、信号機、交通標識又は車線等の検出に用いられる。 The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.
 なお、図37には、撮像部12101ないし12104の撮影範囲の一例が示されている。撮像範囲12111は、フロントノーズに設けられた撮像部12101の撮像範囲を示し、撮像範囲12112,12113は、それぞれサイドミラーに設けられた撮像部12102,12103の撮像範囲を示し、撮像範囲12114は、リアバンパ又はバックドアに設けられた撮像部12104の撮像範囲を示す。例えば、撮像部12101ないし12104で撮像された画像データが重ね合わせられることにより、車両12100を上方から見た俯瞰画像が得られる。 Note that FIG. 37 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.
 撮像部12101ないし12104の少なくとも1つは、距離情報を取得する機能を有していてもよい。例えば、撮像部12101ないし12104の少なくとも1つは、複数の撮像素子からなるステレオカメラであってもよいし、位相差検出用の画素を有する撮像素子であってもよい。 At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple imaging elements, or an imaging element having pixels for detecting phase differences.
 例えば、マイクロコンピュータ12051は、撮像部12101ないし12104から得られた距離情報を基に、撮像範囲12111ないし12114内における各立体物までの距離と、この距離の時間的変化(車両12100に対する相対速度)を求めることにより、特に車両12100の進行路上にある最も近い立体物で、車両12100と略同じ方向に所定の速度(例えば、0km/h以上)で走行する立体物を先行車として抽出することができる。さらに、マイクロコンピュータ12051は、先行車の手前に予め確保すべき車間距離を設定し、自動ブレーキ制御(追従停止制御も含む)や自動加速制御(追従発進制御も含む)等を行うことができる。このように運転者の操作に拠らずに自律的に走行する自動運転等を目的とした協調制御を行うことができる。 For example, the microcomputer 12051 can obtain the distance to each solid object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, and can extract as a preceding vehicle, in particular, the closest solid object on the path of the vehicle 12100 that is traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km/h or faster). Furthermore, the microcomputer 12051 can set the inter-vehicle distance that should be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving, which runs autonomously without relying on the driver's operation.
 例えば、マイクロコンピュータ12051は、撮像部12101ないし12104から得られた距離情報を元に、立体物に関する立体物データを、2輪車、普通車両、大型車両、歩行者、電柱等その他の立体物に分類して抽出し、障害物の自動回避に用いることができる。例えば、マイクロコンピュータ12051は、車両12100の周辺の障害物を、車両12100のドライバが視認可能な障害物と視認困難な障害物とに識別する。そして、マイクロコンピュータ12051は、各障害物との衝突の危険度を示す衝突リスクを判断し、衝突リスクが設定値以上で衝突可能性がある状況であるときには、オーディオスピーカ12061や表示部12062を介してドライバに警報を出力することや、駆動系制御ユニット12010を介して強制減速や回避操舵を行うことで、衝突回避のための運転支援を行うことができる。 For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data to automatically avoid obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines the collision risk, which indicates the risk of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of a collision, it can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by forcibly decelerating or steering the vehicle to avoid a collision via the drive system control unit 12010.
 撮像部12101ないし12104の少なくとも1つは、赤外線を検出する赤外線カメラであってもよい。例えば、マイクロコンピュータ12051は、撮像部12101ないし12104の撮像画像中に歩行者が存在するか否かを判定することで歩行者を認識することができる。かかる歩行者の認識は、例えば赤外線カメラとしての撮像部12101ないし12104の撮像画像における特徴点を抽出する手順と、物体の輪郭を示す一連の特徴点にパターンマッチング処理を行って歩行者か否かを判別する手順によって行われる。マイクロコンピュータ12051が、撮像部12101ないし12104の撮像画像中に歩行者が存在すると判定し、歩行者を認識すると、音声画像出力部12052は、当該認識された歩行者に強調のための方形輪郭線を重畳表示するように、表示部12062を制御する。また、音声画像出力部12052は、歩行者を示すアイコン等を所望の位置に表示するように表示部12062を制御してもよい。 At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured image of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured image of the imaging units 12101 to 12104 and recognizes a pedestrian, the audio/image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio/image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.
 以上、本開示に係る技術が適用され得る車両制御システムの一例について説明した。本開示に係る技術は、以上説明した構成のうち、撮像部12031に適用され得る。車両に搭載される撮像装置では、撮像画像の処理速度を高めることができる。その結果、車両制御システム12000では、車両の衝突回避あるいは衝突緩和機能、車間距離に基づく追従走行機能、車速維持走行機能、車両の衝突警告機能、車両のレーン逸脱警告機能等を、高速で実現できる。 Above, an example of a vehicle control system to which the technology disclosed herein can be applied has been described. The technology disclosed herein can be applied to the imaging unit 12031 of the configurations described above. The imaging device mounted on the vehicle can increase the processing speed of captured images. As a result, the vehicle control system 12000 can quickly achieve functions such as vehicle collision avoidance or collision mitigation, following driving based on the distance between vehicles, vehicle speed maintenance driving, vehicle collision warning, and vehicle lane departure warning.
 以上、実施の形態および変形例、ならびにそれらの具体的な応用例を挙げて本技術を説明したが、本技術はこれらの実施の形態等には限定されず、種々の変形が可能である。  The present technology has been described above by presenting embodiments and modifications, as well as specific examples of their applications, but the present technology is not limited to these embodiments, and various modifications are possible.
 なお、本明細書に記載された効果はあくまで例示であって限定されるものでは無く、また他の効果があってもよい。 Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.
 なお、本技術は以下のような構成とすることができる。以下の構成の本技術によれば、撮像画像の処理速度を高めることができる。
(1)
 複数の受光画素が二次元配置された画素アレイと、
 SNN(Spiking Neural Network)・ハードウェアによって構成された複数のプロセッサが二次元配置されたマルチプロセッサと、
 前記画素アレイと前記マルチプロセッサとに接続された単一階層もしくは複数階層のルータと
 を備えた
 撮像装置。
(2)
 前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素と、前記複数のプロセッサとに接続される
 (1)に記載の撮像装置。
(3)
 前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記単一階層もしくは前記複数階層のルータは、前記デジタル変換器により得られたデジタル信号を前記複数のプロセッサに送信する
 (2)に記載の撮像装置。
(4)
 前記単一階層もしくは前記複数階層のルータは、メモリおよびアービタを有し、前記メモリに保存された複数の前記デジタル信号を、前記アービタの制御に応じて前記複数のプロセッサに順次送信する
 (3)に記載の撮像装置。
(5)
 前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を有し、
 前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路と、前記複数のプロセッサとに接続される
 (1)ないし(4)のいずれか1つに記載の撮像装置。
(6)
 前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記単一階層もしくは前記複数階層のルータは、前記デジタル変換器により得られたデジタル信号を前記複数のプロセッサに送信する
 (5)に記載の撮像装置。
(7)
 前記単一階層もしくは前記複数階層のルータは、メモリおよびアービタを有し、前記メモリに保存された複数の前記デジタル信号を、前記アービタの制御に応じて前記複数のプロセッサに順次送信する
 (6)に記載の撮像装置。
(8)
 前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素から得られた複数の前記デジタル信号を、フレームスタートおよびフレームエンドとともに前記複数のプロセッサに送信し、
 前記プロセッサは、SNNを用いた信号処理を行い、その過程で漏れ積分発火処理を行い、さらに、前記フレームスタートから前記フレームエンドまでの間、前記漏れ積分発火処理において、中間状態の経時的なデクリメントを無効化する
 (5)ないし(7)のいずれか1つに記載の撮像装置。
(9)
 前記複数階層のルータが設けられ、
 前記複数階層のルータは、前記複数の受光画素に対して割り当てられた第1階層の第1ルータと、前記プロセッサごとに1つずつ割り当てられた第2階層の複数の第2のルータとを有し、
 前記第1のルータは、各前記第2のルータと接続される
 (1)に記載の撮像装置。
(10)
 前記第1のルータは、各受光画素と、各前記第2のルータとに接続される
 (9)に記載の撮像装置。
(11)
 前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
 前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (10)に記載の撮像装置。
(12)
 前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を前記第1のグループごとに1つずつ有し、
 前記第1のルータは、前記読み出し回路と、各前記第2のルータとに接続される
 (9)に記載の撮像装置。
(13)
 前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
 前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (12)に記載の撮像装置。
(14)
 前記第2のルータは、当該第2のルータに対応する前記プロセッサ内のニューロンの動作状態についてのデータを前記第1のルータに送信し、
 前記第1のルータは、前記第2のルータからの前記データに基づいて、前記受光画素から得られたデータの送信先を決定する
 (12)または(13)に記載の撮像装置。
(15)
 前記複数階層のルータが設けられ、
 前記複数階層のルータは、前記複数の受光画素が複数の第1のグループに分割されるとともに前記複数のプロセッサが複数の第2のグループに分割されるときに、一組の前記第1のグループおよび前記第2のグループごとに1つずつ割り当てられた第1階層の複数の第1のルータと、一組の前記第1のグループおよび前記第2のグループごとに複数割り当てられた第2階層の複数の第2のルータとを有し、
 前記複数の第2のルータは、前記プロセッサごとに1つずつ割り当てられる
 (1)に記載の撮像装置。
(16)
 各前記第1のルータは、対応する前記第1のグループの各前記受光画素と、対応する前記第2のグループの各前記第2のルータに接続され、さらに、対応する前記第1のグループに隣接する複数の前記第1のルータに接続され、
 各前記第2のルータは、対応する前記第1のルータと、対応する前記プロセッサとに接続され、さらに、隣接する複数の前記第2のルータに接続される
 (15)に記載の撮像装置。
(17)
 前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
 前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (16)に記載の撮像装置。
(18)
 前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を前記第1のグループごとに1つずつ有し、
 各前記第1のルータは、対応する前記第1のグループの前記読み出し回路と、対応する前記第2のグループの各前記第2のルータとに接続され、さらに、対応する前記第1のグループに隣接する複数の前記第1のルータに接続され、
 各前記第2のルータは、対応する前記第1のルータと、対応する前記プロセッサとに接続され、さらに、隣接する複数の前記第2のルータに接続される
 (15)に記載の撮像装置。
(19)
 前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
 前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (18)に記載の撮像装置。
(20)
 前記第2のルータは、当該第2のルータに対応する前記プロセッサ内のニューロンの動作状態についてのデータを前記第1のルータに送信し、
 前記第1のルータは、前記第2のルータからの前記データに基づいて、前記受光画素から得られたデータの送信先を決定する
 (15)ないし(19)のいずれか1つに記載の撮像装置。
(21)
 前記単一階層のルータが設けられ、
 前記単一階層のルータは、前記複数の受光画素が複数の第1のグループに分割されるときに、前記第1のグループごとに1つずつ割り当てられた複数の第1のルータを有し、
 前記複数の第1のルータは、前記プロセッサごとに1つずつ割り当てられる
 (1)に記載の撮像装置。
(22)
 各前記第1のルータは、対応する前記第1のグループの各前記受光画素と、対応する前記プロセッサに接続され、さらに、隣接する複数の前記第1のルータに接続される
 (21)に記載の撮像装置。
(23)
 前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (22)に記載の撮像装置。
(24)
 前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を有し、
 各前記第1のルータは、対応する前記第1のグループの前記読み出し回路と、対応する前記プロセッサに接続され、さらに、隣接する複数の前記第1のルータに接続される
 (21)に記載の撮像装置。
(25)
 前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のルータは、前記デジタル変換器の動作状態に基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
 (24)に記載の撮像装置。
(26)
 前記第1のルータは、当該第1のルータに対応する前記プロセッサ内のニューロンの動作状態に基づいて制御データを生成し、生成した前記制御データを画素アレイに送信し、
 前記画素アレイは、前記第1のルータからの前記制御データに基づいて、各前記受光画素からのデータ出力の要否を決定し、出力要と決定された前記受光画素から得られたデータを前記第1のルータに送信する
 (21)ないし(25)のいずれか1つに記載の撮像装置。
(27)
 前記複数の受光画素が形成された第1のチップと、
 前記マルチプロセッサおよび前記単一階層もしくは前記複数階層のルータのルータが形成された第2のチップと
 を備え、
 前記第1のチップは、前記受光画素ごとに1または複数の第1のパッド電極を有し、
 前記第2のチップは、前記受光画素ごとに1または複数の第2のパッド電極を有し、
 前記第1のチップと前記第2のチップとは、前記第1のパッド電極と前記第2のパッド電極とを互いに重ね合わせて積層される
 (1)ないし(26)のいずれか1つに記載の撮像装置。
(28)
 前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のパッド電極は、前記第1のチップにおいて、受光面とは反対側の表面に設けられ、前記デジタル変換器により得られたデジタル信号が出力される配線に接続され、
 前記第2のパッド電極は、前記第2のチップの表面に設けられ、前記単一階層もしくは前記複数階層のルータの入力端に接続される
 (27)に記載の撮像装置。
(29)
 前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を有し、
前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
 前記第1のパッド電極は、前記第1のチップにおいて、受光面とは反対側の表面に設けられ、前記デジタル変換器により得られたデジタル信号が出力される配線に接続され、
 前記第2のパッド電極は、前記第2のチップの表面に設けられ、前記単一階層もしくは前記複数階層のルータの入力端に接続される
 (27)に記載の撮像装置。
The present technology can be configured as follows: According to the present technology configured as follows, it is possible to increase the processing speed of captured images.
(1)
a pixel array in which a plurality of light receiving pixels are arranged two-dimensionally;
A multi-processor in which multiple processors configured with SNN (Spiking Neural Network) hardware are arranged in two dimensions;
a single-layer or multi-layer router connected to the pixel array and the multiprocessor.
(2)
The imaging device according to (1), wherein the single-layer or multi-layer router is connected to the plurality of light receiving pixels and the plurality of processors.
(3)
The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The imaging device according to (2), wherein the single-layer or multi-layer router transmits the digital signals obtained by the digital converter to the processors.
(4)
The imaging device described in (3), wherein the single-layer or multi-layer router has a memory and an arbiter, and sequentially transmits the digital signals stored in the memory to the processors under control of the arbiter.
(5)
the pixel array has a readout circuit that reads out the light receiving pixels row by row,
The imaging device according to any one of (1) to (4), wherein the single-level or multilevel router is connected to a readout circuit that reads out the light receiving pixels row by row, and to the processors.
(6)
The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The imaging device according to (5), wherein the single-layer or multi-layer router transmits the digital signals obtained by the digital converter to the processors.
(7)
The imaging device according to (6), wherein the single-layer or multi-layer router has a memory and an arbiter, and sequentially transmits the digital signals stored in the memory to the processors under control of the arbiter.
(8)
the single-layer or multi-layer router transmits the digital signals obtained from the light-receiving pixels together with a frame start and a frame end to the processors;
The imaging device described in any one of (5) to (7), wherein the processor performs signal processing using an SNN, performs leaky integration and firing processing in the process, and further disables time-dependent decrement of an intermediate state in the leaky integration and firing processing from the frame start to the frame end.
(9)
The multiple hierarchical routers are provided,
the plurality of hierarchical routers include a first router in a first hierarchy assigned to the plurality of light receiving pixels, and a plurality of second routers in a second hierarchy assigned to each of the processors,
The imaging device according to any one of claims 1 to 5, wherein the first router is connected to each of the second routers.
(10)
The imaging device according to (9), wherein the first router is connected to each of the light receiving pixels and each of the second routers.
(11)
The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The first router transmits data about the operational status of the digital converter to each of the second routers;
The imaging device described in (10), wherein the second router determines a destination of the digital signal obtained by the digital converter based on data regarding an operating state of the digital converter obtained from the first router.
(12)
the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis;
The imaging device according to (9), wherein the first router is connected to the readout circuit and each of the second routers.
(13)
The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The first router transmits data about the operational status of the digital converter to each of the second routers;
The imaging device described in (12) above, wherein the second router determines a destination of the digital signal obtained by the digital converter based on data regarding an operating state of the digital converter obtained from the first router.
(14)
The second router transmits data on the operating state of the neuron in the processor corresponding to the second router to the first router;
The imaging device according to (12) or (13), wherein the first router determines a destination of the data obtained from the light receiving pixels based on the data from the second router.
(15)
The multiple hierarchical routers are provided,
the multiple hierarchical routers include a first hierarchical plurality of first routers each assigned to a pair of the first group and the second group when the light receiving pixels are divided into a plurality of first groups and the processors are divided into a plurality of second groups, and a second hierarchical plurality of second routers each assigned to a pair of the first group and the second group;
The imaging device according to any one of claims 1 to 5, wherein the plurality of second routers are assigned to each of the processors.
(16)
Each of the first routers is connected to each of the light receiving pixels in the corresponding first group and each of the second routers in the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
The imaging device according to (15), wherein each of the second routers is connected to a corresponding one of the first routers and a corresponding one of the processors, and is further connected to a plurality of adjacent second routers.
(17)
The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The first router transmits data about the operational status of the digital converter to each of the second routers;
The imaging device described in (16) above, wherein the second router determines a destination of the digital signal obtained by the digital converter based on data regarding an operating state of the digital converter obtained from the first router.
(18)
the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis;
Each of the first routers is connected to the readout circuit of the corresponding first group and to each of the second routers of the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
The imaging device according to (15), wherein each of the second routers is connected to a corresponding one of the first routers and a corresponding one of the processors, and is further connected to a plurality of adjacent second routers.
(19)
The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The first router transmits data about the operational status of the digital converter to each of the second routers;
The imaging device according to (18), wherein the second router determines a destination of the digital signal obtained by the digital converter based on data on an operating state of the digital converter obtained from the first router.
(20)
The second router transmits data on the operating state of the neuron in the processor corresponding to the second router to the first router;
The imaging device according to any one of (15) to (19), wherein the first router determines a destination of the data obtained from the light receiving pixels based on the data from the second router.
(21)
The single-layer router is provided,
the single-layer router includes a plurality of first routers each assigned to each of the first groups when the plurality of light receiving pixels are divided into a plurality of first groups;
The imaging device according to any one of claims 1 to 5, wherein the plurality of first routers are assigned to each of the processors.
(22)
The imaging device described in (21), wherein each of the first routers is connected to each of the light receiving pixels of the corresponding first group and to a corresponding processor, and is further connected to a plurality of adjacent first routers.
(23)
The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The imaging device according to (22), wherein the first router determines a destination of the digital signal obtained by the digital converter based on data on an operating state of the digital converter.
(24)
the pixel array has a readout circuit that reads out the light receiving pixels row by row,
The imaging device according to (21), wherein each of the first routers is connected to the readout circuit of the corresponding first group and to the corresponding processor, and is further connected to a plurality of adjacent first routers.
(25)
The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
The imaging device according to any one of claims 24 to 30, wherein the first router determines a destination of the digital signal obtained by the digital converter based on an operation state of the digital converter.
(26)
The first router generates control data based on an operation state of a neuron in the processor corresponding to the first router, and transmits the generated control data to a pixel array;
The imaging device described in any one of (21) to (25), wherein the pixel array determines whether or not data output is required from each of the light receiving pixels based on the control data from the first router, and transmits data obtained from the light receiving pixels determined to require output to the first router.
(27)
a first chip on which the plurality of light receiving pixels are formed;
a second chip on which the multiprocessor and the single-layer or multi-layer router are formed;
the first chip has one or a plurality of first pad electrodes for each of the light receiving pixels;
the second chip has one or a plurality of second pad electrodes for each of the light receiving pixels;
The imaging device according to any one of (1) to (26), wherein the first chip and the second chip are stacked with the first pad electrode and the second pad electrode overlapping each other.
(28)
The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
the first pad electrode is provided on a surface of the first chip opposite to a light receiving surface, and is connected to a wiring through which a digital signal obtained by the digital converter is output;
The imaging device according to (27), wherein the second pad electrode is provided on a surface of the second chip and is connected to an input end of the single-layer or multi-layer router.
(29)
the pixel array has a readout circuit that reads out the light receiving pixels row by row,
The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
the first pad electrode is provided on a surface of the first chip opposite to a light receiving surface, and is connected to a wiring through which a digital signal obtained by the digital converter is output;
The imaging device according to (27), wherein the second pad electrode is provided on a surface of the second chip and is connected to an input end of the single-layer or multi-layer router.
 本出願は、日本国特許庁において2022年12月23日に出願された日本特許出願番号第2022-207044号を基礎として優先権を主張するものであり、この出願のすべての内容を参照によって本出願に援用する。 This application claims priority based on Japanese Patent Application No. 2022-207044, filed on December 23, 2022 in the Japan Patent Office, the entire contents of which are incorporated herein by reference.
 当業者であれば、設計上の要件や他の要因に応じて、種々の修正、コンビネーション、サブコンビネーション、および変更を想到し得るが、それらは添付の請求の範囲やその均等物の範囲に含まれるものであることが理解される。 Those skilled in the art may conceive of various modifications, combinations, subcombinations, and variations depending on design requirements and other factors, and it is understood that these are within the scope of the appended claims and their equivalents.

Claims (19)

  1.  複数の受光画素が二次元配置された画素アレイと、
     SNN(Spiking Neural Network)・ハードウェアによって構成された複数のプロセッサが二次元配置されたマルチプロセッサと、
     前記画素アレイと前記マルチプロセッサとに接続された単一階層もしくは複数階層のルータと
     を備えた
     撮像装置。
    a pixel array in which a plurality of light receiving pixels are arranged two-dimensionally;
    A multi-processor in which multiple processors configured with SNN (Spiking Neural Network) hardware are arranged in two dimensions;
    a single-layer or multi-layer router connected to the pixel array and the multiprocessor.
  2.  前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素と、前記複数のプロセッサとに接続される
     請求項1に記載の撮像装置。
    The imaging device according to claim 1 , wherein the single-layer or multi-layer router is connected to the plurality of light-receiving pixels and the plurality of processors.
  3.  前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
     前記単一階層もしくは前記複数階層のルータは、前記デジタル変換器により得られたデジタル信号を前記複数のプロセッサに送信する
     請求項2に記載の撮像装置。
    The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
    The imaging device according to claim 2 , wherein the single-layer or multi-layer router transmits the digital signals obtained by the digital converter to the processors.
  4.  前記単一階層もしくは前記複数階層のルータは、メモリおよびアービタを有し、前記メモリに保存された複数の前記デジタル信号を、前記アービタの制御に応じて前記複数のプロセッサに順次送信する
     請求項3に記載の撮像装置。
    The imaging device according to claim 3 , wherein the single-layer or multi-layer router has a memory and an arbiter, and sequentially transmits the digital signals stored in the memory to the processors under control of the arbiter.
  5.  前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を有し、
     前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路と、前記複数のプロセッサとに接続される
     請求項1に記載の撮像装置。
    the pixel array has a readout circuit that reads out the light receiving pixels row by row,
    The imaging device according to claim 1 , wherein the single-level or multi-level router is connected to a readout circuit that reads out the light-receiving pixels on a row-by-row basis, and to the processors.
  6.  前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
     前記単一階層もしくは前記複数階層のルータは、前記デジタル変換器により得られたデジタル信号を前記複数のプロセッサに送信する
     請求項5に記載の撮像装置。
    The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
    The imaging device according to claim 5 , wherein the single-layer or multi-layer router transmits the digital signals obtained by the digital converter to the processors.
  7.  前記単一階層もしくは前記複数階層のルータは、メモリおよびアービタを有し、前記メモリに保存された複数の前記デジタル信号を、前記アービタの制御に応じて前記複数のプロセッサに順次送信する
     請求項6に記載の撮像装置。
    The imaging device according to claim 6 , wherein the single-layer or multi-layer router has a memory and an arbiter, and sequentially transmits the digital signals stored in the memory to the processors under control of the arbiter.
  8.  前記単一階層もしくは前記複数階層のルータは、前記複数の受光画素から得られた複数の前記デジタル信号を、フレームスタートおよびフレームエンドとともに前記複数のプロセッサに送信し、
     前記プロセッサは、SNNを用いた信号処理を行い、その過程で漏れ積分発火処理を行い、さらに、前記フレームスタートから前記フレームエンドまでの間、前記漏れ積分発火処理において、中間状態の経時的なデクリメントを無効化する
     請求項5に記載の撮像装置。
    the single-layer or multi-layer router transmits the digital signals obtained from the light-receiving pixels together with a frame start and a frame end to the processors;
    The imaging device according to claim 5 , wherein the processor performs signal processing using an SNN, performs leaky integration and firing processing in the process, and further disables decrement over time of an intermediate state in the leaky integration and firing processing from the frame start to the frame end.
  9.  前記複数階層のルータが設けられ、
     前記複数階層のルータは、前記複数の受光画素に対して割り当てられた第1階層の第1ルータと、前記プロセッサごとに1つずつ割り当てられた第2階層の複数の第2のルータとを有し、
     前記第1のルータは、各前記第2のルータと接続される
     請求項1に記載の撮像装置。
    The multiple hierarchical routers are provided,
    the plurality of hierarchical routers include a first router in a first hierarchy assigned to the plurality of light receiving pixels, and a plurality of second routers in a second hierarchy assigned to each of the processors,
    The imaging device according to claim 1 , wherein the first router is connected to each of the second routers.
  10.  前記第1のルータは、各受光画素と、各前記第2のルータとに接続される
     請求項9に記載の撮像装置。
    The imaging device according to claim 9 , wherein the first router is connected to each of the light receiving pixels and each of the second routers.
  11.  前記受光画素は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
     前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
     前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
     請求項10に記載の撮像装置。
    The light receiving pixel has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
    The first router transmits data about the operational status of the digital converter to each of the second routers;
    The imaging device according to claim 10 , wherein the second router determines a destination of the digital signal obtained by the digital converter based on data on an operating state of the digital converter acquired from the first router.
  12.  前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を前記第1のグループごとに1つずつ有し、
     前記第1のルータは、前記読み出し回路と、各前記第2のルータとに接続される
     請求項9に記載の撮像装置。
    the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis;
    The imaging device according to claim 9 , wherein the first router is connected to the readout circuit and each of the second routers.
  13.  前記受光画素もしくは前記読み出し回路は、前記受光画素での光検出により得られた信号をデジタル化するデジタル変換器を有し、
     前記第1のルータは、前記デジタル変換器の動作状態についてのデータを各前記第2のルータに送信し、
     前記第2のルータは、前記第1のルータから取得した、前記デジタル変換器の動作状態についてのデータに基づいて、前記デジタル変換器により得られたデジタル信号の送信先を決定する
     請求項12に記載の撮像装置。
    The light receiving pixel or the readout circuit has a digital converter that digitizes a signal obtained by light detection at the light receiving pixel,
    The first router transmits data about the operational status of the digital converter to each of the second routers;
    The imaging device according to claim 12 , wherein the second router determines a destination of the digital signal obtained by the digital converter based on data on an operating state of the digital converter acquired from the first router.
  14.  前記複数階層のルータが設けられ、
     前記複数階層のルータは、前記複数の受光画素が複数の第1のグループに分割されるとともに前記複数のプロセッサが複数の第2のグループに分割されるときに、一組の前記第1のグループおよび前記第2のグループごとに1つずつ割り当てられた第1階層の複数の第1のルータと、一組の前記第1のグループおよび前記第2のグループごとに複数割り当てられた第2階層の複数の第2のルータとを有し、
     前記複数の第2のルータは、前記プロセッサごとに1つずつ割り当てられる
     請求項1に記載の撮像装置。
    The multiple hierarchical routers are provided,
    the multiple hierarchical routers include a first hierarchical plurality of first routers each assigned to a pair of the first group and the second group when the light receiving pixels are divided into a plurality of first groups and the processors are divided into a plurality of second groups, and a second hierarchical plurality of second routers each assigned to a pair of the first group and the second group;
    The imaging device according to claim 1 , wherein the plurality of second routers are assigned to each of the processors one by one.
  15.  各前記第1のルータは、対応する前記第1のグループの各前記受光画素と、対応する前記第2のグループの各前記第2のルータに接続され、さらに、対応する前記第1のグループに隣接する複数の前記第1のルータに接続され、
     各前記第2のルータは、対応する前記第1のルータと、対応する前記プロセッサとに接続され、さらに、隣接する複数の前記第2のルータに接続される
     請求項14に記載の撮像装置。
    Each of the first routers is connected to each of the light receiving pixels in the corresponding first group and each of the second routers in the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
    The imaging device according to claim 14 , wherein each of the second routers is connected to a corresponding one of the first routers and a corresponding one of the processors, and is further connected to a plurality of adjacent second routers.
  16.  前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を前記第1のグループごとに1つずつ有し、
     各前記第1のルータは、対応する前記第1のグループの前記読み出し回路と、対応する前記第2のグループの各前記第2のルータとに接続され、さらに、対応する前記第1のグループに隣接する複数の前記第1のルータに接続され、
     各前記第2のルータは、対応する前記第1のルータと、対応する前記プロセッサとに接続され、さらに、隣接する複数の前記第2のルータに接続される
     請求項14に記載の撮像装置。
    the pixel array includes a readout circuit for each of the first groups, the readout circuit being configured to read out the light receiving pixels on a row-by-row basis;
    Each of the first routers is connected to the readout circuit of the corresponding first group and to each of the second routers of the corresponding second group, and is further connected to a plurality of the first routers adjacent to the corresponding first group;
    The imaging device according to claim 14 , wherein each of the second routers is connected to a corresponding one of the first routers and a corresponding one of the processors, and is further connected to a plurality of adjacent second routers.
  17.  前記単一階層のルータが設けられ、
     前記単一階層のルータは、前記複数の受光画素が複数の第1のグループに分割されるときに、前記第1のグループごとに1つずつ割り当てられた複数の第1のルータを有し、
     前記複数の第1のルータは、前記プロセッサごとに1つずつ割り当てられる
     請求項1に記載の撮像装置。
    The single-layer router is provided,
    the single-layer router includes a plurality of first routers each assigned to each of the first groups when the plurality of light receiving pixels are divided into a plurality of first groups;
    The imaging device according to claim 1 , wherein the plurality of first routers are assigned to each of the processors one by one.
  18.  各前記第1のルータは、対応する前記第1のグループの各前記受光画素と、対応する前記プロセッサに接続され、さらに、隣接する複数の前記第1のルータに接続される
     請求項17に記載の撮像装置。
    The imaging device according to claim 17 , wherein each of the first routers is connected to each of the light receiving pixels in the corresponding first group and to a corresponding one of the processors, and is further connected to a plurality of adjacent first routers.
  19.  前記画素アレイは、前記複数の受光画素に対して行ごとに読み出しを行う読み出し回路を有し、
     各前記第1のルータは、対応する前記第1のグループの前記読み出し回路と、対応する前記プロセッサに接続され、さらに、隣接する複数の前記第1のルータに接続される
     請求項17に記載の撮像装置。
    the pixel array has a readout circuit that reads out the light receiving pixels row by row,
    The imaging device according to claim 17 , wherein each of the first routers is connected to the readout circuit of a corresponding one of the first groups and to a corresponding one of the processors, and is further connected to a plurality of adjacent first routers.
PCT/JP2023/040305 2022-12-23 2023-11-08 Imaging device WO2024135137A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-207044 2022-12-23
JP2022207044 2022-12-23

Publications (1)

Publication Number Publication Date
WO2024135137A1 true WO2024135137A1 (en) 2024-06-27

Family

ID=91588490

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/040305 WO2024135137A1 (en) 2022-12-23 2023-11-08 Imaging device

Country Status (1)

Country Link
WO (1) WO2024135137A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021013048A (en) * 2019-07-03 2021-02-04 公立大学法人会津大学 Spiking neural network by 3d network on-chip
WO2021210389A1 (en) * 2020-04-14 2021-10-21 ソニーグループ株式会社 Object recognition system and electronic equipment
JP2022509754A (en) * 2018-11-01 2022-01-24 ブレインチップ,インコーポレイテッド Improved spiking neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022509754A (en) * 2018-11-01 2022-01-24 ブレインチップ,インコーポレイテッド Improved spiking neural network
JP2021013048A (en) * 2019-07-03 2021-02-04 公立大学法人会津大学 Spiking neural network by 3d network on-chip
WO2021210389A1 (en) * 2020-04-14 2021-10-21 ソニーグループ株式会社 Object recognition system and electronic equipment

Similar Documents

Publication Publication Date Title
US11582406B2 (en) Solid-state image sensor and imaging device
US11950009B2 (en) Solid-state image sensor
US11336860B2 (en) Solid-state image capturing device, method of driving solid-state image capturing device, and electronic apparatus
US11632510B2 (en) Solid-state imaging device and electronic device
US20210218923A1 (en) Solid-state imaging device and electronic device
US11760267B2 (en) Imaging system
WO2020080383A1 (en) Imaging device and electronic equipment
WO2022130888A1 (en) Image capturing device
US20240236519A1 (en) Imaging device, electronic device, and light detecting method
US11381764B2 (en) Sensor element and electronic device
US11463647B2 (en) Imaging device, imaging system, and imaging method
WO2024135137A1 (en) Imaging device
WO2018139187A1 (en) Solid-state image capturing device, method for driving same, and electronic device
WO2020105301A1 (en) Solid-state imaging element and imaging device
WO2022054742A1 (en) Image capturing element, and image capturing device
US20240177485A1 (en) Sensor device and semiconductor device
US20230308779A1 (en) Information processing device, information processing system, information processing method, and information processing program
KR20240035570A (en) Solid-state imaging devices and methods of operating solid-state imaging devices
WO2021157263A1 (en) Imaging device and electronic apparatus
WO2020090459A1 (en) Solid-state imaging device and electronic equipment
US20240078803A1 (en) Information processing apparatus, information processing method, computer program, and sensor apparatus
WO2023189279A1 (en) Signal processing apparatus, imaging apparatus, and signal processing method
WO2018211985A1 (en) Imaging element, method for controlling imaging element, imaging device, and electronic apparatus
WO2024135094A1 (en) Photodetector device and photodetector device control method
WO2023243222A1 (en) Imaging device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23906508

Country of ref document: EP

Kind code of ref document: A1