All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning

2.1 Principle of operation

The operating principle of our device is illustrated in Figure 1.

Figure 1:

Operating principle of the all-optical ReLU function using a nonlinear photonic waveguide. (A) For positive inputs with phase of ϕ _2ω = +π/2, the phase relationship between the signal and bias is 2ϕ _ω − ϕ _2ω = π/2, which causes SHG that depletes ω and amplifies 2ω. (B) For negative inputs, ϕ _2ω = −π/2, the phase relationship 2ϕ _ω − ϕ _2ω = 3π/2 → −π/2 causes DOPA that amplifies ω and depletes 2ω.

We encode the signal information into the coherent optical field of pulses centered at frequency 2ω, with positive values represented by ϕ _2ω = +π/2 phase states, and negative values represented by ϕ _2ω = −π/2 phase states. By co-propagating the signal pulses with bias pulses centered at frequency ω, with fixed input power and phase at ϕ _ω = +π/2, we can induce different nonlinear optical effects for the two possible ϕ _2ω signal phases depending on the value of the phase relationship 2ϕ _ω − ϕ _2ω . For the positive signal values with phase ϕ _2ω = +π/2, the phase relationship yields 2ϕ _ω − ϕ _2ω = +π/2. This induces second harmonic generation (SHG), which is a χ ⁽²⁾ nonlinear optical process that converts two photons of frequency ω into a photon of frequency 2ω, hence depleting ω and amplifying 2ω. Conversely, for the negative signal values with phase ϕ _2ω = −π/2, the phase relationship yields 2ϕ _ω − ϕ _2ω = 3π/2 → −π/2. This induces degenerate optical parametric amplification (DOPA), which is the inverse process of SHG that converts a photon of frequency 2ω into two photons of frequency ω, hence depleting 2ω and amplifying ω. By judiciously choosing the length and bias power, we can achieve the desired shape of the ReLU function. We emphasize that our approach utilizes coherent parametric processes which allows us to implement both positive and negative values (i.e. the information is encoded in the field amplitude), unlike previous optical [7, 13, 17, 19, 23], [24], [25] and optoelectronic methods [11, 14], [15], [16, 21, 26], [27], [28], [29] based on incoherent absorption processes that can only implement positive values (i.e. the information is encoded in the optical power).

2.2 Device design

To implement the χ ⁽²⁾-based ReLU function, we use a periodically poled thin-film lithium niobate (PPLN) nanophotonic waveguide that exploits the strong and instantaneous χ ⁽²⁾ optical nonlinearity of lithium niobate and tight spatial confinement of the waveguide modes to enhance the nonlinearity [30]. Additionally, careful qausi-phase matching and dispersion engineering enables ultra-broadband and low-energy interactions over mm-long propagation lengths, further enhancing the nonlinear optical processes using femtosecond laser pulses [31], [32], [33]. Images of the device are shown in Figure 2. The PPLN nanophotonic waveguide is L = 2.5 mm long and was fabricated on a 700-nm thick X-cut MgO-doped lithium niobate thin-film on 2-μm thick SiO₂ with lithium niobate substrate by dry etching with Ar⁺ plasma, achieving smooth ridge side-walls with slant angle of θ ≈ 60^° as shown in Figure 2A. The waveguide was electrically poled with a period of 5.17 μm, as shown in Figure 2B, to ensure efficient SHG and DOPA. Dispersion engineering of the fundamental TE mode of the ridge waveguide, shown in Figure 2C, allows for negligible group velocity mismatch and group velocity dispersion of ω and 2ω pulses centered at 1045 and 2090 nm, respectively. This enforces good temporal overlap of the pulses over the entire PPLN propagation length. The ideal parameters found from simulation were a ridge top width of w = 1700 nm and etch-depth of h = 350 nm. See [33] for further details about fabrication and dispersion engineering of PPLN nanophotonic waveguides.

Figure 2:

Images of the PPLN nanophotonic waveguide. (A) Scanning electron microscope image of the ridge waveguide. (B) Two-photon absorption microscope image of the PPLN ferroelectric domains with poling period of 5 μm. (C) Simulated electric field distributions of the fundamental TE modes at 1045 nm (2ω) and 2090 nm (ω).

3.1 Femtojoule ReLU function

The measured response of the all-optical ReLU is shown in Figure 3. The nonlinear function given by the PPLN was measured using a free-space chip characterization setup. The source at 1045 nm (signal) was a Yb:fiber mode-locked laser producing 75-fs long pulses at a 250-MHz repetition rate (Menlo Systems Orange). The same laser pumped a homemade degenerate optical parametric oscillator to generate the pulses at 2090 nm (bias). The 2ω and ω pulses were coupled into and out of the PPLN using reflective objectives focused on the waveguide facets. Finally, the relative phase of the 2ω signal and ω bias was set using a delay arm, and the power varied using a tunable attenuator. See Supplementary Section 1 for further details about the experimental setup.

Our experimental results show good agreement with the ideal ReLU function (R ² = 0.9895), and demonstrates energy-efficient signal pulse energies in the regime of femtojoules per activation. Note that the important feature of the function is its nonlinear shape, since scaling/shifting the horizontal/vertical directions can be accomplished with linear optical transformations. In theory, the ideal ReLU function requires an arbitrarily long PPLN and low bias pulse energy. However, in practice we must choose the bias pulse energy so as to best approximate the ReLU function given our fixed device length. Thus, there are small discrepancies around E _2ω (0) = 0, since neither the SHG nor DOPA processes sufficiently saturate at the ultra-low energies. The maximum cutoff pulse energy is determined by the onset of supercontinuum generation from strong back-conversion processes, which undesirably degrades the pulse shape. To verify that the expected device response matches our physical picture of the operating principle, we also performed nonlinear pulse propagation simulations of the PPLN nanophotonic waveguide. See Supplementary Section 3 for more details about the simulation methods.

Figure 3:

Output signal pulse energy versus input signal pulse energy for both negative and positive inputs. There is good agreement between the ideal ReLU function (dashed black line), simulation (dashed blue line) and experimental results (red circles) for a bias pulse energy of E _ω (0) = 270 fJ, and signal pulse energies of femtojoules per activation.

Remarkably, we show that the PPLN nanophotonic waveguide can also approximate other commonly used variants of the ReLU function, simply by tuning the bias pulse energy. For example, the Exponential Linear Unit (ELU) defined as ELU(x) = x if x > 0 and ELU(x) = exp(x) − 1 if x < 0, which has been shown to outperform the ReLU function in certain cases [34], is achieved using a bias pulse energy of E _ω (0) = 450 fJ as shown in Figure 4A. In addition, we also implement the Gaussian Error Linear Unit (GELU) defined as GELU(x) = xΦ(x) where Φ(x) is the Gaussian cumulative distribution using a bias pulse energy of E _ω (0) = 910 fJ as shown in Figure 4B. The GELU function is used extensively in Transformer networks for natural language processing, which are regularly amongst the largest deep learning models [35]. Thus, our all-optical PPLN nanophotonic waveguide implementation gains greater real-world applicability by being compatible with a wide range of existing deep learning models, especially the largest models where energy efficiency is paramount. Indeed, compatibility has been problematic in previous implementations of optical [7, 17, 23], [24], [25] and optoelectronic [11, 14, 15, 26, 29] nonlinear activation functions, which do not reflect the most commonly used functions in digital electronic neural networks. By alleviating this problem, we expand the potential functionality of ONNs by avoiding the need to train new specialized models.

Figure 4:

Other variants of the ReLU function can be approximated by tuning the bias pulse energy. For example, the (A) ELU function using bias pulse energy of E _ω (0) = 450 fJ and (B) GELU function using bias pulse energy of E _ω (0) = 910 fJ. Ideal function curves are shown by the dashed black lines, and experimental results with red circles.

3.2 Ultrafast time response

Ideally, the time per activation should be near-instantaneous due to the ultrafast χ ⁽²⁾ nonlinearity in lithium niobate. However, in practice, the response time is limited by the finite phase-matching bandwidth as well as non-zero group velocity mismatch, group velocity dispersion, and higher-order dispersion terms. To determine the response time of the device, we used the pump–probe technique commonly used to characterize all-optical switches [32, 36, 37] (see Supplementary for more details). In this case, the pump pulse is the ω pulse and the probe pulse is the 2ω pulse. We measured the ultrafast ReLU dynamics by varying the time delay between the ω and 2ω pulses at a fixed pulse energy. Figure 5 shows the intensity envelope of the pump–probe signal as the time delay is varied as well as the autocorrelation of the input ω pulse.

Figure 5:

Pump–probe ultrafast timing measurements of the ReLU dynamics. The autocorrelation (yellow circles shifted vertically for clarity) of the input ω pulse is well-explained by a Gaussian profile (purple line) with FWHM of (56.4 ± 1.5) fs. The pump–probe signal obtained at a fixed pulse energy (blue circles) is fit (orange line) by convolving the input autocorrelation with exponential growth and decay for positive and negative time delays, respectively. The best fit yields a rise time of (18.9 ± 1.9) fs and a fall time of (28.4 ± 1.1) fs.

The input autocorrelation is well-explained by a Gaussian profile with FWHM of (56.4 ± 1.5) fs. We extract the characteristic rise and fall times by fitting the pump-probe signal with exponential growth and decay functions for positive and negative time delays, respectively, convolved with the input autocorrelation. The best fit yields a rise time of (18.9 ± 1.9) fs and a fall time of (28.4 ± 1.1) fs. This implies that the characteristic response time of the ReLU dynamics is (47.3 ± 3.0) fs, thus verifying that the ultrafast optical nonlinearity is responsible for the ReLU response, and ruling out the possibility of any slower optical nonlinearities such as photorefractive or thermo-optic effects. Therefore, we can reasonably regard the 2ω signal pulse length of ∼75 fs as the time per activation for the all-optical ReLU. We note that better dispersion engineering can lead to even faster activation times.

3.3 Simulated deep learning performance

One distinct advantage of our approach is that, unlike previous all-optical [19] and optoelectronic [21] nonlinear activation functions, it can faithfully reproduce the ideal ReLU function, which uses both positive and negative values. Therefore, we can leverage the large number of existing pretrained deep learning models that use the ReLU function (or its variants) for nonlinear activations. Although ONNs have been demonstrated that accurately reproduce linear operations such as matrix multiplication and convolution, the use of atypical nonlinear activation functions in the optical domain has required the training of new custom deep learning models [38, 39]. To improve upon this, we simulated the performance of the all-optical ReLU function when used as part of a pretrained convolutional neural network (CNN) for the prototypical task of MNIST handwritten digits image classification [40]. The MNIST dataset contains 28 × 28 pixels gray-scale images of handwritten digits with 50,000 training samples and 10,000 test samples. We used a standard CNN architecture (see Supplementary Section 5 for full details) containing convolutional layers and ideal ReLU layers followed by a fully-connected layer and softmax classification output. The pretrained CNN achieved an ideal test accuracy of 99.13%. Next, the ideal ReLU layers were replaced with custom layers representing the experimentally measured ReLU response (after proper shifting/scaling) without changing any of the other layers. This caused a slight drop in test accuracy to 98.8% due to the slight deviations between the experimentally measured and ideal ReLU functions. To remedy this, the CNN was then fine-tuned by training for only 2 epochs (the CNN sees each sample once per epoch) to regain the ideal pretrained model accuracy of 99.13% as shown in Figure 6. Fine-tuning is necessary for any analog hardware implementation due to unavoidable fabrication errors, noise and other nonidealities encountered [41]. Note that this method requires far less time compared to previous proposals for training new custom ONN models, which required >25 training epochs [38, 39]. Therefore, our all-optical ReLU provides the missing link to allow ONNs to take advantage of existing pretrained models. We note that the softmax classification layer is yet to be faithfully implemented in an ONN which accounts for a small portion of the computation compared to the convolutions, matrix multiplications and ReLU nonlinear activations.

Figure 6:

Simulated deep learning performance of the experimentally measured all-optical ReLU function for MNIST handwritten digits image classification. (A) A pretrained CNN was used where the ideal ReLU layers are replaced with custom layers representing the experimentally measured ReLU response (after shifting/scaling) then fine-tuned by training for 2 epochs (batch size of 128) to improve the test accuracy (blue line) back to the ideal pretrained model accuracy (dashed black line). (B) Confusion matrix on the MNIST task for the final network, which achieved 99.13% test accuracy.

4.1 Comparison of energy and time per activation

In this section, we compare the PPLN nanophotonic waveguide to other optical [13, 17, 19, 23], optoelectronic [11, 14], [15], [16, 21, 26], [27], [28], [29], analog electronic [42], [43], [44], and digital electronic [45] nonlinear activation functions to demonstrate the state-of-the-art performance of our device. In this case, the appropriate figure of merit is the energy-time product, which properly accounts for both the energy consumed and time taken per activation. To quantify the energy per activation, we follow the convention in [39], as being the energy needed to generate a 50% change in the power transmission with respect to the transmission with null input. In this case, our device has an energy per activation of ∼16 fJ. The bias pulse energy is not included since it is not destroyed and can, at least theoretically, be reused for many signal pulses. This is because the bias pulse is not dissipated as heat, unlike the case often encountered for absorption-based processes. Assuming perfect phase-matching and that positive/negative values occur equally likely, then the bias pulse should be amplified/deamplified equally likely by the processes of DOPA/SHG, respectively. The time per activation is given by the signal pulse width of ∼75 fs, owing to the near-instantaneous χ ⁽²⁾ nonlinearity of lithium niobate as explained in Section 3.2. Therefore, we achieve an energy-time product of 1.2 × 10^–27 J s. The energy and time per activation of our device is compared to other experimental demonstrations in Figure 7.

Figure 7:

Comparison of energy and time per activation of this work (red star) to previous all-optical (red circle), optoelectronic (blue square), analog electronic (green diamond), and digital electronic (magenta triangle) nonlinear activation functions. The numeric labels show reference numbers and dashed black lines show the energy-time product contours.

We attempted to consider device-level metrics wherever possible to provide a fair comparison, however, we acknowledge that this was not always possible for nonlinear activations as part of complete networks since fan-out and cascadability constraints impose additional energy and time costs. Despite this, the outstanding metrics of our device represents a significant breakthrough for optical nonlinear activation functions. For state-of-the-art digital electronics, such as the NVIDIA A100 Tensor Core GPU [46] based on 7-nm process node [47], we generously assume that the ReLU function consumes ∼1 fJ per activation, and occurs in a single 1 GHz clock cycle. We see that, although our device still has an order of magnitude greater energy per activation, the time per activation is four orders of magnitude faster. Hence, we achieve an energy-time product that is three orders of magnitude better than state-of-the-art digital electronics. Our numerical simulations (Supplementary Section 3) predict that the PPLN nanophotonic waveguide can realistically achieve a ReLU-like response with sub-femtojoule energy per activation. This would even surpass the energy efficiency of state-of-the-art digital electronics. We attribute the discrepancy between our experimental results and the theoretically predicted limits for the energy scale to the imperfect phase-matching and fabrication error of our device. It is worth mentioning how these device-level metrics potentially translate to those of complete neural networks. In this case, additional system-level energy costs such as laser wall-plug efficiency and transport losses can significantly increase the effective activation energy. However, we note that the same is also true in digital electronics such as GPUs where electrical data movement energy costs can exceed the actual switching energy by several orders of magnitude [48].

4.2 Potential network architectures

So far, we have demonstrated how PPLN nanophotonic waveguides can implement all-optical, ultrafast, energy-efficient nonlinear activation functions, which forms only one building block of a full neuron. In this section, we briefly discuss how our device can be integrated into a complete ONN architecture. Interestingly, DOPA and SHG are theoretically noiseless amplification/deamplification processes. Therefore, the all-optical ReLU function should not contribute additional noise to a photonic neural network. In principle, the all-optical ReLU is compatible with most existing ONN architectures that can accurately implement linear operations such as matrix multiplication and convolutions. However, in practice, the speed bottleneck will likely be the encoding of information into the required coherent optical amplitudes. In this case, PPLN nanophotonic waveguides can be monolithically integrated with high-speed electro-optic modulators in thin-film lithium niobate, demonstrated to achieve bandwidths beyond 100 GHz [49]. Furthermore, the light sources can also be integrated on-chip using thin-film lithium niobate optical parametric oscillators [50]. Therefore, all the fundamental building blocks needed for a complete ONN in thin-film lithium niobate already exist. Given the rapid increases in scalability of thin-film lithium niobate photonics, we are confident that a complete ONN can be demonstrated in the near-future. One potential approach is to use Mach–Zehnder interferometer meshes [18] or photonic tensor cores with waveguide cross-bar arrays [20] to implement the linear matrix multiplications, then cascaded into PPLN nanophotonic waveguides to perform nonlinear activations. Another promising method is to use a time-multiplexed architecture similar to ones demonstrated for coherent Ising machines [51] or photonic reservoir computers [14, 15]. See Supplementary Section 6 for more detailed descriptions and schematics of potential integrated lithium niobate nanophotonic neural networks for deep learning.

A valid concern is harnessing the full capabilities of the all-optical ReLU function. It is challenging to fully exploit the ultrafast time response of the nonlinear optical processes since current interfacing electronics is currently limited to GHz bandwidths [48]. However, this should not automatically preclude the use of ultrafast nonlinear optics for optical computing. For example, coherent Ising machines [51] and optical signal processing [52], which require optical input and optical output, are prime candidates for near-term applications. In the future, all-optical computing hardware using such parametric ultrafast nonlinear activation functions may operate with THz clock rates. Crucially, the all-optical ReLU is cascadable since DOPA/SHG are inherently energy-conserving, i.e. the output is sufficiently energetic to serve as the input trigger for at least one other neuron. If multiple outputs are desired, i.e. fan-out, then intermediate amplification is needed, which can be provided by the same type of PPLNs demonstrated. Therefore, in principle, the bottleneck of optoelectronic conversion and analog-to-digital conversion can be bypassed.