This special issue collects eight contributions, selected among many submitted papers after a multi-cycle review process, which are representative of the recent advances on real-time multi-dimensional image processing algorithms and architectures. The published works come from both academia and industry R&D sites in Europe, North America, Asia and Africa. These works show that more and more applications involve multi-dimensional image processing (multi- and hyperspectral imaging, multi-camera vision, multi-frame imaging, etc.), such as remote sensing, surveillance, computer vision and image/video enhancement for avionic and automotive.

The proposed works address algorithmic and architectural optimizations trying to overcome the major limit of multi-dimensional imaging, which is their high computational and memory requirement.

Several complex computing platforms exist, such as graphical processing unit (GPU), multi-core digital signal processors (DSP) or hardware designs based on field programmable gate array (FPGA) and/or application specific integrated circuit (ASIC). Recently, GPUs have attracted a great deal of attention, since they have demonstrated their potential for multi-dimensional imaging applications, although limited to specific algorithms. Nonetheless, their use has to be tailored to various constraints such as power consumption, size and weight.

The first work by S. Sanchez et al., entitled “Fast determination of the number of endmembers for real-time hyperspectral unmixing on GPUs”, addresses the challenging problem in spectral unmixing applications of how to determine the number of endmembers in a given scene. Spectral unmixing is a very important task for remotely sensed hyperspectral data exploitation. It involves identifying a set of spectrally pure components (called endmembers) and their associated per-pixel coverage fractions (called abundances). Several automatic techniques exist for this purpose, including the ones based on the virtual dimensionality (VD) concept or on the hyperspectral signal identification by minimum error (HySime). Due to the complexity and high dimensionality of hyperspectral scenes, these techniques are computationally expensive. The paper by S. Sanchez et al. introduces new fast implementations of VD and HySime using commodity graphics processing units. The proposed parallel implementations are validated in terms of accuracy and computational performance, showing significant speedups with respect to optimized serial implementations. The newly developed implementations are integrated in a fully operational unmixing chain which exhibits real-time performance with respect to the time that the hyperspectral instrument takes to collect the image data.

The use of GPU platform for real-time image enhancement is also addressed by the work of Yuan-Kai Wang et al., entitled “A CUDA-enabled parallel algorithm for accelerating Retinex”, which is an image restoration approach used to restore the original appearance of an image. The paper presents a GPURetinex algorithm, which is a data parallel algorithm accelerating a modified center/surround retinex with GPU/CUDA. Particularly, the GPURetinex algorithm exploits the massively parallel threading and heterogeneous memory hierarchy of a GPU to improve efficiency. Special care has been devoted to solve issues such as irregular memory access and block size for data partition. Experimental results conducted on GT200 GPU and CUDA 3.2 showed that the GPURetinex can gain 74 times acceleration, compared with an SSE-optimized single-threaded implementation on Core2 Duo™ for the images with 16 Mpixel resolution. The proposed method also outperforms the parallel retinex implemented with the nVidia™ Performance Primitives library.

The problem of partitioning and parallelization of multi-dimensional vision algorithms on multi-core processing platforms or on GPU has been addressed also by Tobias Duckworth et al. in their work entitled “Parallel processing for real-time 3D reconstruction from video streams”. The target of the work is real-time reconstruction of 3D scenes from multiple video streams to achieve commodity telepresence systems capable of communicating both what someone looks like and what, within the technology joined space, they are looking at.

The theme of GPU-accelerated image processing systems is addressed also in the work by Liang Wang et al. entitled “Real-Time Stereo Using Approximated Joint Bilateral Filtering and Dynamic Programming”. In this work, the authors present a stereo framework that operates in real time while still estimating high-quality depth information for live stereo video sequences. The proposed algorithm combines edge-preserving cost-volume filtering and dynamic programming optimization. The use of a color and distance weighted cost aggregation window in the vertical direction significantly reduces ‘‘streaking’’ artifacts. Experimental results show that it is among the best performing real-time stereo algorithms in terms of both disparity estimation accuracy and efficiency. In addition, an approximation for the 2D bilateral aggregation is developed, which leads to a fully GPU-accelerated implementation to achieve two orders of speed-up compared to the state of the art. This simplified approach can produce reasonably accurate disparity maps in real time. Looking into the future, optimizing DP using SIMD instructions will further improve the speed performance.

While multi-core or GPU-based real-time implementations are examples of software-oriented solutions, where real-time or near real-time processing of computing intensive tasks are achieved at the cost of power consumption, other works, such as that of M. Turturici et al., address the propel of mixed hardware–software solutions for low-power consumption multi-image computing. In their work entitled “Low-power DSP system for real-time correction of fish-eye cameras in automotive driver assistance applications” where they propose an embedded DSP system acquiring wide FOV using multiple fish-eye cameras and correcting their distortion in real time. The paper proposes a solution that can be easily adapted to different types of lenses and cameras, up to four cameras, and meets real-time constraints with a power budget within 100 mW and a board size of a few cm2.

As a new frontier for high data rate multi-dimensional image processing, directly realized in hardware, the work by Farnood Merrikh-Bayat, et al., entitled “Memristive fuzzy edge detector”, presents the use of a multi-layer neuro-fuzzy computing system, based on the memristor crossbar structure. The paper also introduces a new concept called the fuzzy minterm, and shows how it can be used to extract edges from grayscale images. One of the advantages of the proposed memristive fuzzy edge detector (implemented in analog form), compared to other commonly used edge detectors, is that it can be implemented in parallel form, which makes it a powerful device for real-time applications.

The last two papers deal with algorithmic-level optimization of multi-dimensional image processing tasks.

The work by M. A. Mahraz et al., entitled “Motion estimation using the fast and adaptive bidimensional empirical mode decomposition”, proposes a new technique to estimate the optical flow in multi-frame vision systems. The proposed approach is based on the FABEMD (fast and adaptive bidimensional empirical mode decomposition) with the aim of improving the well-known pyramidal algorithm of Lucas and Kanade (LK) which, in principle, utilizes two consecutive frames extracted from a video sequence to determine a dense optical flow. The proposed algorithm uses the FABEMD method to decompose each of the two considered frames into several BIMFs (bidimensional intrinsic mode functions) that are matched in number and properties. Thus, to compute the optical flow, the LK algorithm is applied to each of the two matching BIMFs, which belong to the same mode of the decomposition. Although the implementation does not use an iterative refinement, the results show that the proposed approach is less sensitive to noise and provides improved motion estimation with a reduction of computing time compared to iterative methods.

Finally the work by A. Rossi et al., entitled “RX architectures for real-time anomaly detection in hyperspectral images” concerns the design of computationally efficient anomaly detection (AD) algorithms for hyperspectral images to assure real-time or near real-time processing. In the field of hyperspectral image processing, AD is a deeply investigated method with the goal of finding objects in the image that are anomalous with respect to the background. In many operational scenarios, detection, classification and identification of anomalous spectral pixels have to be performed in real time to quickly furnish information for decision-making. In this work, a sub-class of AD algorithms is considered, i.e., those algorithms aimed at detecting small rare objects that are anomalous with respect to their local background. Among such techniques, one of the most established is the Reed–Xiaoli (RX) algorithm, which is based on a local Gaussian assumption for background clutter and locally estimates its parameters by means of the pixels inside a window around the pixel under test (PUT). In this work, the state of the art RX real-time oriented techniques have been improved using a linear algebra-based strategy to efficiently update the inverse covariance matrix thus avoiding its computation and inversion for each pixel of the hyperspectral image. The proposed strategy has been deeply discussed pointing out the benefits introduced on the two analyzed architectures in terms of overall number of elementary operations required. The results show the benefits of the new strategy with respect to the original architectures.

In conclusion, the guest editors hope that the selected papers will provide the readers with interesting samples of present research on algorithms and architectures for real-time multi-dimensional image processing applications. They are very grateful to the reviewers who provided valuable comments and suggestions to improve the quality of the accepted papers.