Soft Delivery: Survey on a New Paradigm for Wireless and Mobile Multimedia Streaming

This work provides a comprehensive survey of soft delivery schemes, including an overview on existing techniques, extension for immersive experiences, and future research directions. Some existing studies have focused on soft delivery schemes, which are shown in Figure 1, with a brief description of the related topics and key techniques. Although there are some survey papers [9, 16, 17, 18, 19, 20] related to the video delivery over wireless channels, all of the survey papers deal with digital-based approaches. To the best of our knowledge, this survey is the first to introduce methodologies and approaches for soft and Hybrid Digital–Analog (HDA) delivery to transmit high-quality image and video signals via unstable and diverse wireless and mobile channel environments. The main contributions of this study are summarized as follows:

This survey identifies that soft delivery works particularly well in untethered immersive experiences including free viewpoint video, VR, and point cloud. Soft delivery yields better reconstruction quality compared with HEVC-based delivery schemes by integrating the energy compaction, power allocation, and overhead reduction techniques discussed in existing studies. In addition, soft delivery can realize high-quality and adaptive delivery of Deep Neural Network (DNN) models and tactile information over wireless channels. Such delivery will help to realize future services including Federated Learning (FL) and untethered XR applications.

The remainder of this article is organized as follows:

One of the major issues in wireless video delivery is sending high-quality videos within the considerably limited capacity of wireless links. For this purpose, standardized digital video compression is carried out for video frames in conventional video delivery schemes [21, 22, 23], as shown in Figure 2(a) to remove redundancy among video frames. In particular, H.264/Advanced Video Coding (AVC) [24], H.265/HEVC, and H.266/VVC standards are typical video coding standards for generating a compressed bitstream from video frames. In such video encoders, pixel values in each video frame are divided into blocks and transformed into frequency domain coefficients by using the Discrete Cosine Transform (DCT) or discrete sine transform, and then non-uniformly quantizing the coefficients according to a quantization parameter. A large quantization parameter indicates a larger quantization step, leading to a smaller bit rate. Finally, the quantized coefficients are compressed by an entropy coder, which removes statistical redundancy in the coefficients. Variable length coding is widely deployed for entropy coding because of its efficiency and simplicity.

After passing through the digital video compression, the bitstream is then passed to the wireless transmission system in sequence. There are typically two ways to deliver the bitstream: Internet Protocol (IP) and non-IP networks. For the IP networks, the sender uses DASH [9, 10] or Real-Time Transport Protocol [25] for the bitstream. Although the IP-based protocols can deliver the content via the deployed IP networks, they cause transmission delays to packetize and de-packetize the bitstream [26]. For the non-IP networks, the sender passes the bitstream to the PHY (physical layer) for transmissions. The existing studies related to the non-IP network schemes reported that such schemes reduce the delay compared with the IP-based schemes, and thus the non-IP schemes are applicable for low-delay applications including untethered XR experiences and cooperative and competitive gaming.

This survey mainly discusses the content delivery over non-IP networks. In such non-IP schemes, channel coding is first used for the bitstream to protect against channel errors. For example, binary convolutional codes and low-density parity checks are widely used as forward error correction in Wi-Fi systems. The coded bitstream is then mapped onto in-phase and quadrature (I and Q) components using digital modulation formats such, as quadrature phase shift keying and m-ary QAM. In both wireless and mobile networks, a combination of modulation formats and different channel coding rates, for example, 1/2 and 3/4, is defined in the Modulation and Coding Scheme (MCS). According to the measured wireless channel SNRs, the sender adapts its MCS value to maximize the link data rate. At the receiver end, bit errors may occur in channel-coded bits owing to effective noise and/or fading effects. The receiver then tries to reconstruct video frames from the received bits using inverse procedures (i.e., demodulation, channel decoding, and video decoding).

If the measured wireless channel quality is stable during video transmission, conventional digital-based schemes can provide high-quality video frames for users. However, the channel quality of each user fluctuates over time owing to a combination of user mobility, multipath propagation, and obstacle shadow. Table 1 lists three critical issues regarding the video quality of the digital-based schemes because of the channel quality fluctuations: cliff, leveling, and staircase effects.

To prevent the cliff, leveling, and staircase effects in wireless video delivery, a pioneer soft delivery work, namely SoftCast, was proposed by Jakubczak et al. [11, 12, 13]. The block diagram of SoftCast is illustrated in Figure 2(b). The design of SoftCast is based on a simple principle that ensures the transmitted signal samples are linearly related to the original pixel values. This principle naturally enables a sender to satisfy multiple receivers with diverse channel qualities, as well as a single receiver, where different packets experience different channel qualities.

The sender first takes a Group of Pictures (GoP) and uses a full-frame 3D-DCT as the decorrelation transform. The DCT frames are then divided into N small rectangular blocks of transformed coefficients called chunks. The coefficients in each chunk are then scaled to match the transmission power constraints. Specifically, the scaling coefficients are chosen to minimize the reconstruction Mean Square Error (MSE). Walsh–Hadamard transform is then applied to the scaled chunks for power normalization across the chunks to provide packet loss resilience. This process transforms the chunks into slices. Each slice is a linear combination of all scaled chunks. Finally, the coefficients in the slices are directly mapped to the I and Q components in a pseudo-analog manner for transmission. Here, channel coding operations are skipped for the coefficients.

Figure 4(a) and (b) show the conventional digital-based modulation (i.e., 16-QAM) and pseudo-analog modulation proposed in SoftCast. Conventional modulation modulates channel-coded bits to produce real-value digital samples that are transmitted to the channel. For example, 16-QAM modulation takes sequences of four bits and maps each sequence to a complex I/Q number, as shown in Figure 4(a). After modulation, the wireless PHY of the sender transmits the mapped complex numbers to the receiver. Because of the broadcast nature of the wireless medium, multiple receivers hear the transmitted samples but with different noise levels. A receiver with a low channel SNR can distinguish only the quadrant of the transmitted sample and hence can decode only the two bits of the transmitted sample. In this case, these bit errors may cause a collapsed signal reconstruction during digital video decoding.

In contrast to the existing modulation design, SoftCast outputs the real values of the DCT coefficients that are already coded for error protection. The pseudo-analog modulation in Figure 4(b) directly maps pairs of the scaled DCT coefficients to the I and Q of the digital signal samples. Figure 4(c) shows the distribution of the analog-modulated symbols of the test video sequence of “Akiyo” with the resolution of common intermediate format [37]. As shown in Figure 4(c), we find that the pseudo-analog modulation of the DCT coefficients follows a bivariate Lorentzian distribution as follows:where a and b represent the fitting parameters. From least-squares fitting based on the distribution of pseudo-analog symbols of the test video sequence and the bivariate Lorentzian function, the best fitting parameters are a of 0.001 and b of 0.24. As mentioned earlier, multiple receivers hear the transmitted samples under different channel SNRs. Although the transmitted samples are distorted according to their SNR, the receiver regards the received samples as scaled DCT coefficients. The sender does not need to estimate the channel condition, and the noise level in the received samples faithfully reflects the instantaneous channel condition [38]. Thus, pseudo-analog modulation ensures that the received video quality is proportional to the instantaneous channel quality. Consequently, this process avoids all cliff, leveling, and staircase effects.

In parallel, SoftCast sends an amount of data, referred to as metadata, for signal reconstruction. These metadata consist of the mean and variance of each transmitted chunk as well as a bitmap. The mean of each chunk is used to obtain the chunks’ approximate zero-mean distributions by subtracting the mean of all pixels in each chunk [39]. The variance of each chunk is used to find the per-chunk scaling factors that minimize the reconstruction error. The bitmap indicates the positions of the discarded chunks into the GoP. When the available channel bandwidth for SoftCast is less than the required bandwidth, SoftCast discards chunks with less energy. Specifically, when the available and required bandwidths for SoftCast are M chunks and \(N (\gt M)\) chunks, respectively, SoftCast discards lower-energy \(M-N\) chunks to meet the bandwidth requirement. On the receiver side, these discarded chunks are replaced by null values. The discarded chunks are registered as a bitmap and then compressed using run-length encoding. Metadata are strongly protected and transmitted in a robust way (e.g., BPSK modulation format with a low-rate channel code) to ensure correct delivery and decoding.

At the receiver side, a Minimum Mean Square Error (MMSE) decoder is used to estimate the content of the chunks to counteract channel noise. The MMSE provides a high-quality estimate of the DCT coefficients by leveraging the knowledge of the statistics of the DCT coefficients (i.e., chunk variance) as well as the statistics of the channel noise. Using the metadata, the denoised chunks are properly reassembled and undergo an inverse 3D-DCT, thereby providing the corresponding GoP. In principle, the preceding design and performance do not affect the content types (i.e., on-demand or live content). Regardless of the content type, SoftCast provides adaptive video delivery based on the channel quality between the sender and each receiver.

Table 2 shows the strengths, limitations, and use cases of the digital-based and soft delivery schemes. The digital-based delivery schemes are well suited for point-to-point communications over time-invariant channels. In addition, the buffering cost is relatively low compared to the soft delivery schemes because the required storage size for the compressed bitstream is small. Soft delivery schemes perform well in time-varying and diverse channels. As well, soft delivery schemes are low delay because they do not need to perform an expensive motion search for compression, making soft delivery schemes preferable for delay-sensitive applications. However, they require a high cost for modulation and demodulation in the PHY because modifying the PHY in both sender and receiver is required for the pseudo-analog modulation.

In summary, the typical use cases of digital-based delivery schemes are streaming services over wide area networks and HTTP adaptive streaming. In contrast, soft delivery schemes are well suited for delay-sensitive applications such as video broadcasting, Vehicle-to-Everything (V2X) communication, and real-time video surveillance.

In SoftCast, a full-frame 3D-DCT is carried out for the video frames in each GoP to compact the energy of the video signals, and the resulting DCT coefficients are transmitted to the receivers using pseudo-analog modulation. Here, the transmitted analog-modulated symbols are degraded over wireless channels at each receiver. SoftCast should minimize the MSE between the received and transmitted DCT coefficients under the wireless channel SNR to reconstruct the highest-quality video frames at each receiver. For this purpose, SoftCast must design MSE-minimized power allocation and denoising filters (i.e., scaling and inverse scaling operations) for analog-modulated symbols.

Figure 5 illustrates the procedures for obtaining the reconstructed DCT coefficients at the receiver end. SoftCast implements chunk-wise power allocation and filter operations according to the statistics of the chunks and channel conditions. Let \(x_{i}\) denote the ith analog-modulated symbol. Each analog-modulated symbol is scaled by \(g_{i}\) for noise reduction:Here, \(s_{i}\) is the ith DCT coefficient and \(g_{i}\) is the scale factor for the coefficient power allocation. The sender performs optimal power control for \(g_i\) to achieve the highest video quality. Specifically, the best \(g_{i}\) is obtained by minimizing the MSE under the power constraint with the total power budget P as follows:

where \(\mathbb {E}[\cdot ]\) denotes the expectation, \(\hat{x}_{i}\) is an estimate of the transmitted symbol, \({\lambda }_{i}\) is the power of the ith DCT coefficient, N is the number of DCT coefficients, and \(\sigma ^2\) is the receiver noise variance. The near-optimal solution is expressed as

After transmission over the wireless channel, each symbol at the receiver end can be modeled as \(y_{i} = x_{i} + n_i\), where \(y_{i}\) is the ith received symbol and \(n_{i}\) is an effective noise with a variance of \(\sigma ^2\). The receiver extracts DCT coefficients from the I and Q components and reconstructs the coefficients using the MMSE filter [11] as follows:The receiver then obtains the corresponding video sequence using the inverse 3D-DCT for the filter output \(\hat{s}_i\).

In soft delivery schemes via linear mapping (from source signals to channel signals), the reconstruction quality greatly depends on the performance of the energy compaction technique for the source signals. Specifically, the study by Prabhakaran et al. [40] clarified that the performance of soft delivery schemes degrades as the ratio of maximum energy to minimum energy of the source component increases. To yield better quality under both stable and unstable channel conditions, existing studies have adopted different energy compaction techniques listed in Table 3 for the source signals.

Typical solutions are to adopt wavelet-based signal decorrelation methods. Specifically, some studies [41, 42, 43, 44, 45, 46] have adopted a Motion-Compensated Temporal Filter (MCTF), which is a temporal wavelet transform method, to remove inter-frame redundancy by realizing motion compensation in soft delivery. The MCTF recursively decomposes video frames into low- and high-frequency frames according to a predefined level. For example, WaveCast [44] adopted a 3D-Discrete Wavelet Transform (DWT) (i.e., the integration of 2D-DWT and MCTF) to remove temporal and spatial redundancy. Although SoftCast exploits a full-frame 3D-DCT to remove the intra- and inter-frame redundancy for energy compaction, WaveCast can further improve the reconstruction quality by fully exploiting the inter-frame redundancy using motion compensation. A detailed discussion on the effects of other decorrelation methods is presented in the work of Xiong et al. [47, 48]. Trioux et al. [49] also utilized inter-frame redundancy by designing an adaptive GOP size mechanism. It adaptively controlled the GoP size based on shot changes and the spatio-temporal characteristics of the video frames. It then used a full-frame 3D-DCT for energy compaction across the video frames in one GoP.

Another typical solution is to send large energy coefficients as metadata and thus prevent the transmission of such coefficients using pseudo-analog modulation. Lin et al. [50] designed Advanced SoftCast (ASoftCast) to send low-frequency coefficients as the metadata. ASoftCast decomposed the original images into frequency components using 2D-DWT. The frequency component was then divided into two parts: the lowest-frequency sub-band and other sub-bands. The wavelet coefficients in the lowest-frequency sub-band are processed by run-length coding. They are then channel coded and digitally modulated for additional metadata transmissions. The optimized power allocation for the SoftCast scheme in the work of He et al. [51] selected and sent high-energy coefficients as the metadata to reduce the energy of the analog-modulated symbols. These results can assign a high transmission power to low-energy coefficients to improve the received quality. Here, determining the high-energy coefficients for each GoP is computationally complex owing to the use of an exhaustive search. To reduce the computational complexity, Trioux et al. [52] adopted a zigzag scan to select the side information. Other studies [53, 54, 55, 56, 57, 58] divided the video into BL and ELs, which were coded and sent in digital and pseudo-analog ways, respectively. For example, the BL in gradient-based image SoftCast (G-Cast) [57] sent the DC and low-frequency coefficients of the image, whereas the EL extracted and sent an image gradient, which represents the edge portion of the image, using a gradient transform. The receiver then created a final estimation of the image via a gradient-based reconstruction procedure, utilizing both the image gradient at the EL and the low-frequency coefficients provided by the BL.

Other solutions adopted a nonlinear encoder and decoder for source signals to decrease the ratio of the maximum to the minimum energy of the analog-modulated symbols. The typical solution for soft delivery is to introduce coset coding [59, 60], which is a typical technique in distributed source coding. Coset coding partitions the set of possible source values into several cosets and transmits the coset residual codes to the receiver. With the received coset codes and the predictor, the receiver can recover the source value in the coset by choosing the one closest to the predictor. DCast [61, 62, 63, 64] first introduced coset coding for the soft delivery of inter frames. The coset coding in DCast divides each frequency domain coefficient \(s_i\) by a coset step q and obtains the coset residual code \(l_i\) as \(l_i = s_i - \lfloor \frac{s_i}{q} + \frac{1}{2} \rfloor q\), where \(\lfloor \frac{s_i}{q} + \frac{1}{2} \rfloor\) represents the coset index. At this time, the sender only needs to transmit the coset residual code for energy compaction. At the user side, with the received coset residual code \(\hat{l}_i\) and the side information \(\bar{s}_i\) (i.e., the predicted DCT coefficient obtained from the reference video frame), the receiver reconstructs the DCT coefficients by coset decoding. Given the coset residual code \(l_i\), there are multiple possible reconstructions of \(s_i\) that form a coset \(C = \lbrace \hat{l}_i, \hat{l}_i \pm q, \hat{l}_i \pm 2q, \hat{l}_i \pm 3q, \ldots \rbrace\). DCast then selects the coset C that is nearest to the side information \(\bar{s}_i\) as the reconstruction of the DCT coefficient. In this case, the value of each coset step q is crucial for the coding performance of DCast. The value of q is calculated by estimating the noise at the receiver end shown in the work of Fan et al. [63, 64]. However, the reconstruction quality of DCast also depends on the side information quality. If the side information \(\bar{s}_i\) is error prone, the receiver may make wrong decisions with a smaller q. Huang et al. [65] introduced a side information refinement algorithm [66] to refine the side information for the quality enhancement of DCast.

The concept of coset coding has been widely applied in other studies on soft delivery for the same purpose. For example, several works [67, 68, 69, 70, 71] utilized pseudo-coset coding for lower-frequency components and sent the coset index using the digital framework. Here, the residuals in the lowest-frequency components and other frequency components are sent using pseudo-analog modulation. The main difference between coset coding and pseudo-coset coding is the sending of the coset index as additional metadata. The layered coset coding and adaptive coset coding were applied to the soft delivery scheme in the work of Fan et al. [69] and Lv et al. [70], respectively. LayerCast [69] introduced layered coset coding to simultaneously accommodate heterogeneous users with diverse SNRs and bandwidths. The layered coset coding used large to small coset steps to obtain coarse to fine layers from each chunk. The coarse layer (i.e., BL) is sufficient to reconstruct a low-quality DCT chunk for narrowband users, whereas each fine layer (i.e., EL) provides refinement information of the DCT chunk for wideband users. Some works [72, 73, 74] utilized the coset coding for cooperative soft delivery systems (i.e., a three-node relay network). A sender broadcasts the DCT coefficients obtained from the video frames using pseudo-analog modulation to the relay node and the destination node. If the channel quality between the sender and the destination node is higher than a threshold, the destination node reconstructs the video frames from the soft-delivered DCT coefficients. If the channel condition is lower than the threshold, the relay node sends the coset residual code to the destination node, then the destination node reconstructs the video frames using the received coset residual code and the side information obtained from the softly delivered DCT coefficients from the sender.

As mentioned in Section 3.2, the power allocation in SoftCast minimizes the MSE between the original and reconstructed video signals over Additive White Gaussian Noise (AWGN) channels. There are several drawbacks toward adopting SoftCast in practical scenarios: (1) practical wireless channels have more complex characteristics (e.g., fading caused by multipath and impulse noise) than the AWGN channels, and (2) MSE is not an effective index for describing the perceptual fidelity of images/videos. To address the drawbacks related to the power allocation, the existing studies in Table 4 propose the power allocation for practical wireless channels and perceptual considerations.

For the first drawback, the existing studies redesigned the power allocation for practical wireless channels, including fading [75] and frequency-selective fading (i.e., Orthogonal Frequency-Division Multiplexing (OFDM)) [76, 77], impulse noise [84], Multiple-Input and Multiple-Output (MIMO) [78, 79, 80], and MIMO-OFDM channels [81, 82, 83]. Cui et al. [75] designed an optimal power allocation for fading channels. In fading channels, a fading effect (i.e., multiplicative noise) will degrade the reconstruction quality. Although SoftCast assumes that multiplicative noise can be canceled with exact channel estimation at the receiver end, no algorithm can guarantee an error-free channel estimation. In addition to the power allocation design, the authors analyzed the effect of the channel estimation error on the reconstruction quality at the receiver end.

For frequency-selective fading channels, such as OFDM and MIMO-OFDM channels, the key issue is how to match the analog-modulated symbols to the independent subcarriers/subchannels for high-quality image/video reconstruction. Liu et al. [81, 82] observed similarities between the source and channel characteristics and exploited the similarities for subcarrier/subchannel matching. ParCast [81] and the extended version of ParCast\(+\) [82] assigned the more important DCT coefficients to higher gain channel components and allocated power weights for each DCT coefficient with joint consideration of the source and channel for video unicast systems. ECast [83] extended the source and channel matching and power allocation for video multicast systems. For multicast systems, it is necessary to deal with the large overhead of channel feedback from multiple receivers. In ECast, multiple users simultaneously send tone signals for the channel feedback, and the sender receives the superposition of multiple tone signals. Although the sender cannot distinguish each of the channel gains, the weighted harmonic means of channel gains can be obtained from the superposed tone signals. Thus, ECast utilizes the channel gain for the source and channel matching and power allocation.

Other studies solved power allocation problems in modern wireless systems, including Non-Orthogonal Multiple Access (NOMA) [85, 86], underwater acoustic OFDM [87], Unmanned Aerial Vehicle (UAV)-enabled [88], and mmWave lens MIMO systems [89]. For example, in NOMA systems, source signals are coded into the BL and ELs and then transmitted simultaneously through superposition coding. With successive interference cancellation, near users with strong channel gains can decode both BL and EL signals, whereas far users with weak channel gains may only decode BL signals. In the existing studies, both the BL and ELs are analog coded in the work of Jiang et al. [85], whereas the BL and ELs are digital coded and analog coded, respectively, in the work of Wu et al. [86]. They solved the power allocation across the BL and ELs to minimize the distortion for all receivers with heterogeneous channel conditions. In underwater acoustic OFDM [87] and mmWave lens MIMO systems [89], the error behavior differed substantially across channel components, and the channel characteristics showed a similar tendency. They solved the source and channel matching and power allocation problems, which are also discussed in frequency-selective fading channels, to minimize the distortion at the receiver end.

For the second drawback, some studies [90, 91, 92, 93] also redesigned the power allocation with perceptual considerations, including Structural Similarity (SSIM) [90], foveation [91], and saliency [92]. In these studies, determining the perception-aware weights for each source component is challenging. Specifically, in SoftCast, the scaling factor for each coefficient is obtained from its power information to minimize the MSE: \(g_i \varpropto\lambda _i^{-1/4}\). These studies considered the perception-aware weight for the ith coefficient \(w_i\) in the scaling factor to minimize the perceptual distortion as \(g_i \∝_i^{1/4} \lambda _i^{-1/4}\). For this purpose, Zhao et al. [90] demonstrated the relationship between the MSE in the DCT coefficients and the SSIM distortion to obtain the weight for the ith DCT coefficients of all chunks \(w_i\). They found that the weight for the high-frequency coefficients was larger than that for the low-frequency coefficients, which was consistent with the characteristics of the Human Visual System (HVS). FoveaCast [91] introduced the foveation-based HVS [94] and the corresponding HVS-based visual perceptual quality metric, called Foveated Weighted Distortion (FWD), for the optimization objective. For a given foveation point \((f_x, f_y)\) in the pixel and frequency domains, the error sensitivity for each pixel/frequency coefficient at location \((x, y)\) can be defined in the foveation-based HVS. FoveaCast regarded the error sensitivity in the DWT domains as the weight \(w_i\) and performed foveation-aware power allocation. In the work of Hadizadeh [92], visual saliency maps were introduced for the perception-aware power allocation. Saliency maps represent the attended regions in an image when a user watches the image owing to the visual attention mechanism of the human brain. In this case, the weight for the ith pixel \(w_i\) is based on the normalized visual saliency defined from any arbitrary visual saliency model, such as the Itti–Koch–Niebur model [95]. Based on the weight, it allocates considerable transmission power to salient regions to minimize the Eye-Tracking Weighted Mean Square Error (EQMSE).

The source bandwidth of soft delivery schemes depends on the number of transmitted analog-modulated symbols every second (i.e., baud rate). In the aforementioned designs, the source bandwidth is mainly considered sufficient to send all transmitted non-zero analog-modulated symbols over the wireless medium. However, when the channel bandwidth is lower than the source bandwidth, some analog-modulated symbols are discarded at the receiver side. Hence, the loss of the important coefficients (i.e., the low-frequency coefficients) may have a significant impact on the reconstruction quality. Specifically, the expected distortions in soft delivery schemes for single and multiple content owing to the bandwidth constraint under the transmission power constraint are discussed in the work of Liu et al. [111] and He et al. [112, 113], respectively. Some existing studies have adopted different techniques listed in Table 5 to meet the bandwidth constraint. The typical method is to selectively discard the chunks in higher-frequency components to fill the bandwidth [11, 96]. When the sender discards some chunks, the receiver regards all coefficients in the discarded chunks as zeros. Because it needs to send the locations of the discarded chunks to the receiver, SoftCast sends the location information as a bitmap. Although SoftCast assumes equal-size chunks across low- to high-frequency components, Li et al. [96] adopted smaller chunk sizes in high-frequency components to realize a fine-grained control to meet the bandwidth limitation. Another study [97] used bandwidth-reducing Shannon–Kotelnikov (SK) mappings to increase the number of chunks transmitted over bandwidth-constrained channels. The SK mappings are typical N:1 bandwidth-reducing or 1:M bandwidth-expanding non-linear mappings. In this study, 2:1 SK mappings were used to encode several pairs of chunks with less energy to send more chunks with medium energy within the channel bandwidth.

Other studies [98, 99, 100, 101, 102, 103, 104, 105] introduced Compressive Sensing (CS) techniques [114, 115] for soft delivery over bandwidth-constrained wireless channels. Notably, CS is a sampling paradigm that allows the simultaneous measurement and compression of signals that are sparse or compressible in some domains. In general, recovering source signals from compressed signals is impossible because the system is underdetermined. However, if the source signals are sufficiently sparse in some domains, CS theory indicates that the source signals can be reconstructed from the compressed signals by solving the \(\ell _1\) minimization problem. The advantage of CS-based soft delivery is the recovery of chunks in high-frequency coefficients using CS-based signal reconstruction algorithms, such as approximate message passing and iterative thresholding, even though the chunks are discarded at the sender’s end. For high-quality reconstruction, adaptive rate control and reconstruction algorithms are mainly adopted for CS-based soft delivery. For instance, Yami and Hadizadeh [100] adaptively controlled the compression rate based on visual attention (i.e., both the texture complexity and visual saliency) to satisfy the bandwidth constraint while maintaining better perceptual quality. Liu et al. [104] adaptively selected reliable columns from the measurement matrix and compressed source signals using the selected columns. In view of the reconstruction algorithm, Hadizadeh and Bajic [101] designed an adaptive transform for noisy measurement signals to obtain sparser transform coefficients for clean reconstruction. Yin et al. [102] and Tung and Gunduz [103] designed grouping methods for measurement signals to utilize the similarity between video frames for the reconstruction.

Other studies utilized stored images/videos on the cloud to reduce the bandwidth requirement in soft delivery. Specifically, data-assisted communications of mobile images (DAC-Mobi) [106], data-assisted cloud radio access network (DaC-RAN) [107], and knowledge-enhanced mobile video broadcasting (KMV-Cast) schemes [108, 109, 110], which are referred to as data-assisted soft delivery schemes, have been proposed for high-quality image/video transmission. The main contributions of the data-assisted soft delivery schemes are (1) a sender sends a limited number of analog-modulated symbols and (2) the receiver reconstructs images/videos using correlated images (i.e., side information) obtained from a cloud.

In DAC-Mobi [106], successive coset encoders were introduced to divide the DCT coefficients into three layers of bit planes: Most Significant Bits (MSBs) in low-frequency coefficients, MSBs in other frequency coefficients and middle bits, and Least Significant Bits (LSBs). Here, MSBs in low-frequency coefficients and LSBs were transmitted to the receiver in digital and pseudo-analog manners, respectively, whereas MSBs in other frequency coefficients and middle bits were discarded. Based on the received MSB in the low-frequency coefficients, the receiver reconstructs a down-sampled image to retrieve correlated images in the cloud. The retrieved correlated images were used as side information to resolve ambiguity due to discarded bits and reconstruct the entire image. DaC-RAN [107] and the extended version of KMV-Cast [108, 109, 110] adopted Bayesian reconstruction algorithms that utilize correlated images/videos in the cloud as prior information to reduce the required bandwidth for soft delivery. The main difference between the DaC-RAN and KMV-Cast schemes is that the former assumes that the same images/videos exist in the cloud, whereas the latter does not require that the same images/videos exist at the receiver end by designing prior knowledge broadcasting in a digital manner.

The aforementioned studies considered the channel bandwidth to be lower than the source bandwidth. If the channel bandwidth is greater than the source bandwidth, the soft delivery schemes become less efficient. In this case, the soft delivery schemes utilize the extra bandwidth by retransmission. Lin et al. [116] and Tan et al. [117] designed an analog channel coding to use the extra channel bandwidth for quality enhancement. For example, Tan et al. [117] proposed a chaotic function-based analog encoding [118] for soft delivery. Although the existing chaotic function-based analog coding is designed for uniformly distributed sources, the analog coding for Gaussian distributed sources significantly amplifies source signals and thus consumes unnecessary transmission power. They designed a chaotic map function for Gaussian distributed source signals to prevent power increments compared to the input power. MCast [119] also utilized extra bandwidth for quality improvement. As mentioned earlier, the sender can send the source data multiple times if an extra bandwidth is available. In this case, the utilization of extra time slots for quality improvement is a key issue. To overcome this issue, MCast optimized the assignment of the chunks of the DCT coefficients to available channels in multiple time slots to fully exploit the time and frequency diversities.

In contrast to the aforementioned studies, Lan et al. [120] and He et al. [121] dealt with bandwidth variations. When the available bandwidth is less than the expected bandwidth at the sender’s end, some important chunks will not have the opportunity to be transmitted before the playback deadline. They grouped several chunks into a tile and sent the tile with a large variance and high priority to dispatch important coefficients before the playback deadline.

Even when the channel bandwidth is sufficient to send all non-zero analog-modulated symbols, some analog-modulated symbols can be discarded at the receiver side owing to loss-prone wireless channels. Specifically, the packet loss owing to strong fading and interference may have a significant impact on the reconstruction quality if important chunks and coefficients are lost. SoftCast used Walsh–Hadamard transform to redistribute the energy of the source signals across whole packets for resilience against packet loss. However, each packet still contains a large amount of energy, and thus degradation owing to packet losses remains considerable.

To maintain better reconstruction quality in error-prone wireless channels, some related studies [122, 123, 124] have introduced CS techniques (i.e., block-wise CS [125]) for packet loss resilience. The CS technique is suitable for wireless transmission with random packet loss owing to its random measurement. Random measurement considers all packets as of equal importance. In contrast to typical CS techniques, block-wise CS can reduce the storage and computational costs of the reconstruction. A pioneering work on packet loss resilience is the distributed compressed sensing-based multicast scheme (DCS-Cast) [122]. In DCS-Cast, each image is first divided into blocks and the coefficients in each block are randomized using the same measurement matrix across the blocks. One coefficient in every block is packetized to normalize the importance across packets. Even though some packets may be lost over loss-prone wireless channels, the receiver obtains noisy pixel values using the same measurement matrix at the sender and reconstructs the lost pixel values using the CS reconstruction algorithm in the DCT/DWT domains. Because the lost pixel values can be recovered from the reconstruction algorithm, DCS-Cast maintains high image/video quality in loss-prone channels. To further improve the reconstruction quality, multi-scale [123] and adaptive [124] block-wise CS algorithms have been adopted for soft delivery. The multi-scale block-wise CS algorithm [123] decomposes each video frame into a multi-level 2D-DWT and then optimizes the sampling rate for each DWT level according to its importance. However, the adaptive block-wise CS algorithm [124] divides several video frames into one reference frame and subsequent non-reference frames and adaptively determines whether direct or predictive sampling should be used for each block in a non-reference frame. Direct sampling randomizes the signals in the block, whereas predictive sampling calculates the residuals between the blocks in the reference and non-reference frames and randomizes residuals to utilize the inter-frame similarity for the reconstruction.

In soft delivery schemes without chunk division, a sender needs to let the receiver know the power information of all the DCT coefficients to demodulate the signals. For the receiver to carry out the MMSE filtering in Equation (5), the sender needs to transmit \({\lambda }_{i}\) of all coefficients without errors as metadata, which may constitute a large overhead. For example, when the sender transmits eight video frames with a resolution of \(352\times 288\), the sender needs to transmit metadata for all DCT coefficients (i.e., \(352\times 288\times 8 = 811{,}008\) variables in total) to the receiver. This overhead may induce performance degradation owing to the rate and power losses in the transmission of analog-modulated symbols. To reduce the overhead, SoftCast divides the DCT coefficients into chunks and carries out chunk-wise power allocation using an MMSE filter. However, overheads are still high, and chunk division causes performance degradation due to improper power allocation.

To achieve better quality under a low overhead requirement, the related studies can be classified into two types, as shown in Figure 6: sender-side overhead reduction and receiver-side overhead reduction. Studies on the sender-side overhead reduction [126, 127, 128, 129] designed fitting functions to obtain the power information with fewer parameters. In this case, the sender and receiver share the same fitting function in advance and send the parameters as metadata for overhead reduction. Specifically, Song et al. [126] designed a fitting function with four parameters for each chunk, and Xiong et al. [127] designed a log-linear function with two parameters for each chunk. Another study [128] found that equal-size chunk division was not suitable for chunk-wise fitting, and thus an adaptive chunk division (i.e., L-shaped chunk division) was designed for an accurate fitting. In addition, Fujihashi et al. [129] exploited a Lorentzian fitting function with seven parameters based on a Gaussian Markov Random Field (GMRF) for each GoP. The sender-side studies accurately predict the metadata using the fitting function with a limited number of the parameters (i.e., low overhead), whereas the fitting function causes an additional computational cost.

Studies on receiver-side overhead reduction [130, 131] estimate the power information only from the received signals without any additional computational cost at the sender side. The study of Li et al. [130] is a pioneer work to estimate the power information from the received signals. Blind data detection [131] was proposed to decode the received analog-modulated symbols without the power information at the receiver. Specifically, blind data detection uses a zero-forcing estimator and the sign of the received signals to approximate the source signals. One typical issue of the receiver-side overhead reduction is that the reconstruction quality highly depends on the quality of the received signals.

We note that both types of overhead reduction cause quality degradation owing to estimation errors. In the work of Xiong et al. [127], the effect of modeling accuracy on the reconstruction quality in soft delivery was analyzed.

The aforementioned studies mainly discussed performance improvements in theoretical analyses and simulations. Table 6 lists the categories of the existing studies in terms of the performance evaluation. Existing studies [41, 45, 81, 82, 121, 134] used the software-defined radio platform SORA [141] to carry out emulations. In contrast to the simulations, the emulations obtain channel fading and noise trace from SORA to evaluate the performance under real wireless environments.

Some studies implemented a soft delivery scheme on a software-defined radio platform [13, 103, 117, 135] and an Field-Programmable Gate Array (FPGA) [136, 137, 138] to empirically demonstrate the benefits of soft delivery in practical wireless channels. In some works [13, 103, 135], the authors used Universal Software Radio Peripheral (USRP) 2, USRP NI2900, and USRP X310 and GNU Radio for implementation and evaluated the visual quality of soft delivery, respectively. In addition, Tan et al. [117] built an experimental system based on the OpenAirInterface (OAI) platform and self-developed Software Universal Platform (SOUP). They migrated OAI to self-developed SOUP software-defined radio and implemented the proposed scheme based on OAI eMBMS codes. Conversely, other works [136, 137, 138] exploited the Xilinx Virtex7 FPGA for implementation and tested the reconstruction quality as a function of wireless channel SNRs.

Other studies [139, 140] implemented soft delivery on the prototypes of multi-user MIMO (MU-MIMO) and Long-Term Evolution (LTE) systems. For example, in the work of Chen et al. [139], SoftCast is implemented on BUSH, which is a large-scale MU-MIMO prototype that performs scalable beam user selection with hybrid beamforming for phased-array antennas in legacy WLANs. They performed experiments to evaluate the video quality in terms of Peak Signal-to-Noise Ratio (PSNR) and SSIM over a lossy MU-MIMO channel.

SoftCast and other soft delivery schemes mentioned in the previous sections were designed for conventional images and video signals. In modern wireless and mobile communication scenarios, the streaming of immersive content will be a key application for reconstructing 3D perceptual scenes that provide full parallax and depth information for human eyes. The immersive content can be applied to various applications, such as 3 to 6 degrees-of-freedom entertainment, remote device operation, medical imaging, vehicular perception, VR/AR/MR, and simulated training. The typical immersive content includes free viewpoint video [142, 143, 144], 360-degree video, and point cloud [145], and Table 7 lists the features. Even in immersive content, the video frames are compressed in a digital manner, and the compressed bitstream is then channel coded and modulated in sequence. This means that cliff and leveling effects still occur in the streaming of the immersive content owing to the variation in the channel conditions. To prevent cliff, leveling, and staircase effects, some studies have extended soft delivery schemes toward immersive content for future wireless multimedia services.

One of the key advantages of soft delivery schemes for immersive content is that they simplify the optimization problem for image and video quality maximization. In immersive content delivery, the main issue is to maximize the image and video quality considering the user’s perspective. For example, the view synthesis distortion optimization problem and the viewport optimization problem should be solved in free viewpoint video and 360-degree video, respectively. In digital-based delivery schemes, a sender needs to find the best bit and transmission power allocation for video frames. However, it is often cumbersome to derive a solution. Soft delivery schemes simplify the optimization problems by reformulating them into a simple power allocation problem since bit allocation for quantization is not required in soft delivery schemes.

For further quality improvement, the pioneer studies of HDA delivery [182, 183] integrate low-rate digital-based encoding and decoding into soft delivery.

They proposed the superposition of analog-coded and digital-coded symbols to take advantage of conventional digital-based and soft delivery schemes. Specifically, the digital-coded symbols provide the baseline quality of the video frames, whereas the analog-coded symbols enhance the quality of the video frames according to the wireless channel quality. Here, the low-rate digital-based operations can significantly reduce the signal energy of the analog-coded symbols (e.g., the decrement of the ratio of the maximum energy to the minimum energy of the source component). A theoretical study [40] clarified that a lower ratio improves the reconstruction quality of the analog-coded symbols. This means that the amount of quality improvement with the improvement of the wireless channel quality in HDA coding schemes becomes more significant compared to the pure soft delivery schemes. Nonetheless, the integration with digital-based encoding has one drawback: the cliff effect may occur when the decoding of digital-coded symbols fails.

Figure 7 shows an overview of the HDA delivery schemes. HDA delivery schemes consist of the digital and analog coding parts. At the sender side, the video frames are first encoded by the digital video encoder and the digitally coded bitstream is channel coded, modulated, and assigned transmission power by the sender. Meanwhile, the residuals are coded, power assigned, and modulated by the soft delivery scheme. Both outputs from the digital and analog coding parts are superposed and transmitted over wireless channels. In this case, the transmitted signal \(x_i\) is the sum of the BPSK-modulated vector signal \(x^{\langle \mathsf {d}\rangle }_{i}\) and output vector signal of the soft delivery scheme \(x^{\langle \mathsf {a}\rangle }_{i}\) as follows:The BPSK-modulated symbol and the analog-modulated symbol are scaled by \(P_\mathrm{d}\) and \(g_{i}\), respectively:where \(b_{i}\in \mathbb {X} = \left\lbrace \pm 1 \right\rbrace\) is the BPSK-modulated symbol and \(\jmath =\sqrt {-1}\) denotes the imaginary unit. Here, the near-optimal solution of \(g_i\) under the transmission power budget \(P_\mathsf {a}\) is based on Equation (4). We note that the budgets of the transmission power for the digital and analog parts need to satisfy the total power budget \(P_\mathsf {t}\) (i.e., \(P_\mathsf {t} = P_\mathsf {a} + P_\mathsf {d}\)).

At the receiver side, it first decodes the digital-modulated symbols and then obtains the analog-modulated symbols by subtracting the digital-modulated symbols from the received symbols. Finally, the receiver reconstructs the baseline quality of the video frames from the output of the digital part and enhances the video quality by adding the output of the analog part.

A key issue in HDA delivery is the assignment of transmission power to the digital and analog parts [184]. Specifically, the power assigned to the digital part must guarantee the correct decoding of the symbols. By contrast, the digital decoder treats the superimposed analog-modulated symbols \(x^{\langle \mathsf {a}\rangle }_{i}\) as noise. To achieve better decoding performance, the I component of \(x^{\langle \mathsf {a}\rangle }_{i}\) should be kept as small as possible. In the work of Song et al. [185], the authors only select the high-frequency coefficients, which are expected to be very small values for superposition. The remaining low-frequency coefficients are delivered using pseudo-analog modulation. The HDA framework in the work of Tan et al. [186] regards the superposed symbols as three main parts: orthogonal analog symbols, digital symbols, and nonorthogonal analog symbols superimposed onto digital symbols. They designed resource allocation among these three parts to achieve a better balance between lowering interference and improving reconstruction quality. Another study [187] designs a prediction model to describe the relationship between the variance of residuals and the quantization parameter, and determines the optimal transmission power for the analog part, which maximizes the reconstruction quality with the correct decoding of the digital part. The HDA delivery scheme in the work of Zhang et al. [188] treats the imperfect decoding of the digital part and finds the best assignment of the transmission power for the digital and analog parts. This prevents too much power assignment for the digital part to ensure a low bit error rate. In contrast to the aforementioned studies, Liang et al. [189] treat the bandwidth of other digital traffic as hidden resources for HDA video delivery. Specifically, they superimpose the analog-modulated symbols and digital symbols of the other digital traffic to utilize the hidden resource under the constraint that the bit error rate requirement of the other digital traffic is not compromised.

Other studies have redesigned the power allocation in HDA delivery for practical wireless channel environments, including fading [190, 191], OFDM [192, 193], MIMO [194], and relay networks [195]. For example, the power allocation with perfect channel state information is designed in the work of Shen et al. [190], Yahampath [192], and Liu et al. [194], whereas the power allocation with imperfect channel state information is designed in other work by Yahampath [191, 193]. In view of the packet loss resilience in HDA delivery, the study of Fujihashi et al. [196] introduced CS for the residuals.

Other studies [197, 198, 199] extend HDA video delivery for immersive content. Swift [197] considers stereo video delivery and designs a zigzag coding structure for the stereo video to utilize both intra- and inter-view correlations. In the zigzag coding structure, the odd frames in the left view and the even frames in the right view are encoded digitally, and the rest of the frames are encoded in analog. Here, the reconstructions of the digitally coded frames are used as side information to further remove redundant information from the analog-coded frames. Another study [198] extends HDA delivery for MVD videos and solves the view synthesis optimization to yield the best quality from an intermediate virtual viewpoint. HoloCast+ [199] designs HDA delivery for point cloud delivery.

In future work, the recent coding standards such as H.266/VVC, learned video compression [200, 201], and point cloud coding can be used for the digital part. Although they have achieved significant energy compaction and can further improve the quality of the analog part, the recent coding standards require a long coding delay for compression. The tradeoff between coding delay and reconstructed image and video quality is an open question in HDA delivery.

Some recent studies integrate DNN architectures for nonlinear encoding and decoding operations of soft delivery, namely AI-empowered soft delivery. AI-empowered soft delivery schemes utilize Deep Convolutional Neural Networks (DCNNs) [202, 203] and multi-layer perceptron networks for energy compaction, power allocation, and overhead reduction tasks.

The multi-layer perceptron auto-encoder was first adopted to reduce the overhead of soft delivery [204]. Specifically, the proposed encoder obtains a few latent variables from the pixel values, and the proposed decoder decodes the accurate power information from the received latent variables for proper power allocation. The reconstruction quality can be maintained even with only one metadata across one GoP. Another study [205] designs Deep Joint Source-Channel Coding (DJSCC) for the energy compaction of the image and video signals. The DJSCC schemes integrate a DCNN-based auto-encoder into a soft delivery scheme. The proposed encoder directly compresses each image into a limited number of latent variables, and the proposed decoder reconstructs the image from the latent variables. Here, the latent variables are transmitted over wireless channels using pseudo-analog modulation. Even though the latent variables are obtained by nonlinear functions and delivered over wireless channels with a lower SNR, cliff and leveling effects can be prevented via pseudo-analog modulation. Other studies have introduced the DNN architecture for power allocation [206] and decoding operations [207]. The study of Tang et al. [206] uses a YOLO (you-only-look-once) structure [208] to extract the Region of Interest (ROI) and non-ROI parts from each image and then assign unequal transmission power across ROI and non-ROI parts for perceptual quality enhancement. The proposed scheme in the work of Fujihashi et al. [207] integrates DCNN-based image denoising, specifically DIP (deep-image-prior) [209], into soft delivery. The DIP finds linear and nonlinear noise effects for reconstructing clean images from noisy images. The proposed scheme can remove fading and noise effects from the received images using DCNN-based image restoration. Another study [210] introduces the Graph Neural Network (GNN) [211] for wireless point cloud delivery. The GNN is a novel model for graph representation learning that allows the analysis of the irregular geometric structure of graph data. GNN-based auto-encoder [212, 213] was designed to encode 3D point clouds into a limited number of latent variables. One of the benefits of the GNN-based auto-encoder is that it allows graph signal reconstruction from a limited number of latent variables without requiring additional metadata.

DJSCC schemes with the latest neural network architectures have been well studied for further energy compaction in recent years. As a result, they have achieved better image and video quality without cliff, leveling, and staircase effects. However, the existing DJSCC schemes need to deal with the bandwidth heterogeneity among the receivers. Here, how to adaptively improve the image and video quality according to the available bandwidth using the same architecture remains challenging work. In addition, DJSCC schemes have been considered as the fundamental techniques for realizing semantic communication [214, 215] in future wireless and mobile networks. Semantic communication has been envisioned as a new transmission paradigm that delivers semantic meaning rather than a bit stream of transmitted messages. Another challenging issue is to design an optimal DJSCC scheme for the given semantic triple.

Our study and other studies find that soft delivery can be utilized to support various AI architectures. Specifically, many AI-based applications need to exchange trained DNN models between the receiver over wireless networks within a short delay including viewport prediction [216] in untethered XR applications, and dead reckoning in autonomous driving [217, 218] and online gaming services [219]. In recent years, digital-based model compression schemes [220, 221] have been designed and standardized for sharing the trained model over the networks. However, the existing studies found that the cliff and leveling effects occur even in the DNN model transmissions. To prevent both effects, analog modulation is effective in the model transmissions. AirNet [222] adopts analog modulation to deliver the DNN model parameters over wireless networks. Specifically, AirNet directly maps the model parameters to the transmission symbols and sends the analog-modulated symbols via wireless channels. This process avoids the issues mentioned previously, and the model restoration quality faithfully corresponds to the instantaneous channel condition. As well, AirNet adopts SK mappings [97] to reduce the number of transmission symbols for band-limited channels.

In addition, the model parameter transmission is key technique to realize FL [223, 224] over wireless networks. Figure 8(a) shows one typical FL example over wireless networks. The FL is a decentralized learning approach, which trains the model over a federation of distributed learners and an aggregator, to obtain an accurate model even with a limited dataset in each distributed learner. Each learner in the federation uses only locally available data for training. For the training over the distributed learners and aggregator, both learners and the aggregator exchange the model parameters over wireless channels. The existing studies found that the analog modulation-based solutions are efficient for exchanging the model parameters for the FL. AirComp (analog over-the-air computation) [225, 226] is a typical solution for the model parameter transmission in FL. All learners simultaneously send the analog-modulated parameters with channel inversion to the aggregator, and the aggregator can receive the aggregated model parameters from the superimposed waveforms. Although the simultaneous analog transmission can improve the throughput of the model parameter transmission, AirComp requires a precise symbol-level synchronization between the distributed learners. In previous work [227], we proposed the model parameter transmission in a round-robin manner for quasi-asynchronous FL systems. Specifically, Federated AirNet based on HDA delivery integrates low-rate model parameter compression and energy-compact analog modulation.

Figure 8(b) shows the average top-1 classification accuracy of the global model as a function of wireless channel SNR when an available number of transmission symbols is at most 6.0 Msymbols. The number of transmission iterations from 10 learners to the aggregator is 10. We compare Federated AirNet to two state-of-the-art digital and analog approaches: DeepCABAC and AirNet. In the DeepCABAC scheme, the compressed bitstream is channel coded by a half-rate convolutional code with a constraint length of 8 and digitally modulated by BPSK or 4-QAM formats. The AirNet scheme directly maps each of the model parameters onto a transmission symbol in analog modulation. We found that the AirNet and Federated AirNet schemes prevent drastic degradation in accuracy since they do not rely on quantization and entropy coding. In addition, the proposed Federated AirNet scheme yields the best accuracy, in higher-channel SNR regimes, achieving near error-free performance.

Although the soft delivery schemes have the potential for the realization of FL, they cause large energy consumption because they consider the transmission power of each distributed learner as identical. Although each learner can limit the transmission power to reduce energy consumption, the limitation may cause a long delay for convergence and low global model performance. The discussion on the transmission power allocation among the distributed learners will be a key issue for the realization of FL with low energy consumption.

In addition to visual information, multiple sensorial media (mulsemedia) delivery can enhance the quality of immersive experiences in XR applications. Tactile is one of the typical mulsemedia data. Especially, tactile communications can support the untethered and immersive XR applications. In contrast to visual information, the sampling rate of tactile information is relatively high (i.e., above 1,000 Hz) and the delay requirement for tactile communications is strict. In this case, the sender does not retransmit the tactile under the channel quality fluctuation. Although the haptic codec [228, 229], which is defined in the IEEE 1918.1.1 standard, is designed to compress the tactile (e.g., vibrotactile signals), the cliff and leveling effects also occur owing to the channel quality fluctuation. We use the analog modulation with DCT and DWT for the vibrotactile signals to discuss the feasibility of the soft haptic delivery. Figure 9(a) shows an overview of the proposed soft haptic delivery scheme, and Figure 9(b) shows the reconstruction quality of the vibrotactile signal over the soft haptic and the digital-based delivery schemes at the channel SNRs of 10 dB and 20 dB under different available bandwidths. Here, the digital-based delivery scheme uses the BPSK modulation format and controls the quantization parameter to fit the transmission symbols into the available bandwidth, whereas the soft haptic delivery scheme discards high-frequency coefficients for the same purpose. We used “1spike_Probe_-_aluminumGrid_-_fast” as the reference vibrotactile data provided by the IEEE 1918.1.1 standard. From preliminary evaluations, the soft haptic delivery scheme yields better reconstruction quality of the vibrotactile signals irrespective of the available bandwidth.

The soft haptic delivery scheme was designed only for the single vibrotactile sensor and to minimize the MSE between the original and reconstructed vibrotactile signals. However, vibrotactile signals from multiple sensors should be delivered to provide immersive experiences for users. In addition, a psychohaptic model in the work of Noll et al. [228] and Steinbach et al. [229] demonstrated that each frequency band of vibrotactile signals has unequal sensitivity for humans. Here, how to design energy compression for vibrotactile signals from multiple sensors and power allocation to minimize the distortion considering human tactile perception are challenging issues for the realization of haptic delivery.