Optimizing Integrated Features for Hindi Automatic Speech Recognition System

1.1 Feature Integration Review

While human beings can segregate and recognize signals under noisy conditions, it still remains a challenge for an automatic speech recognizer to perform this task accurately in noisy environments. Various methods to enhance speech or to compensate for noise have been applied by researchers to improve the accuracy of ASR systems in adverse conditions. Speech signal filtering prior to the training phase, adapting speech models to include the effects of noise, and using noise robust features are the three categories of classification of these methods. One of the simplest and effective way to deal with noise speech signal is to use noise robust features like Gammatone frequency (GF) cepstral coefficients (GFCCs) instead of using conventional features like linear predictive cepstral coefficients [25], TRAPs (temporal patterns) [15], Mel frequency (MF) cepstral coefficients (MFCCs) [9], perceptual linear prediction (PLP) [14], and wavelets [30], which do not perform accurately in the presence of additive noise.

In Refs. [28] and [34], linear discriminant analysis (LDA) has been applied to integrate MFCC feature vectors with phase features and with voiced-unvoiced features, respectively [35]. MFCC has been integrated with main spectral peak features by using a log-linear model in Ref. [31], and PLP has been integrated with modulation spectrogram features using acoustic posterior probabilities [16]. All these proposed feature integrations significantly reduced the word error rate [16, 28, 31, 34].

In Ref. [35], the recognizer uses three distinct acoustic feature optimizations – vocal tract length-normalized MFCC, PLP coefficients derived from Mel scale (MF-PLP), and MFCC derived from an all-pole magnitude spectrum – by applying LDA and log-linear model integration methods. In Ref. [29], it is shown that GFCC outperforms MF-PLP integration in recognition accuracy and the ROVER (recognizer output voting error reduction) algorithm outperforms the log-linear model and LDA in combining features.

The sequential integration of the features MFCC, gravity centroids, and PLP has been implemented to enhance the accuracy of a Hindi language ASR system for medium-sized vocabulary in specific environments in Ref. [5]. Recently, in Refs. [7] and [11], the MFCC and GFCC features are integrated to improve the accuracy of an automatic speaker recognition system and an ASR system, respectively, in low signal-to-noise ratio (SNR) conditions.

1.2 ASR Optimization Review

Researchers have made various efforts to optimize the front and back end of the ASR system by using different optimization approaches. At the back end, the hidden Markov model (HMM) conventionally uses the Baum-Welch (BW) algorithm that is dependent on correct initial estimation of model attributes. Hence, an incorrect or arbitrary estimation of such parameters does not lead to optimal solutions. The issues of non-linear time wrapping and ideal number of states in HMM topology have been addressed by Kwong et al. in Refs. [18] and [19], respectively, by using genetic algorithms (GAs). In Ref. [33], hybrid combinations of GA-BW and particle swarm optimization (PSO)-BW are compared for continuous HMM optimization. The results conclude that PSO-BW outperforms GA-BW in recognition performance. PSO has also been combined with the Viterbi algorithm to improve the recognition accuracy of the decoding phase of the ASR system [22]. Both the GA and PSO techniques have been applied in the front-end feature extractor module to optimize MFCC filter banks [4]. The results conclude that the ASR performance is improved by using PSO- and GA-optimized filter banks. Also, GA and PSO have been used for integrated feature vector refinement in Ref. [11]. However, in both Refs. [11] and [4], the results conclude that PSO outperforms GA in recognition accuracy. In Ref. [11], a discriminatively trained PSO-optimized feature-based ASR system has an accuracy rate of 5.4%, better than the discriminatively trained GA-optimized feature-based ASR system. In Ref. [4], the ASR system using PSO-optimized filter banks has approximately 3%–5% more accuracy than the ASR system using GA-optimized filter banks. Also, it has been shown that PSO converges at a faster rate and requires less computational time than GA.

The information-sharing idea of the PSO approach is entirely different from techniques like GA and differential evolution. The evolution criteria in PSO searches only for the optimal solution, where only the global best shares information with the other particles. However, in the other two methods, the mechanism of group sharing is used where each particle shares information with each other. One common limitation with PSO is the problem of premature convergence after some iterations, which results in a suboptimal solution. Researchers have used PSO after including a crossover operator in it, and have proposed PSO with crossover (C-PSO) and PSO with quadratic-crossover (Q-PSO) [13, 23].

The proposed work mainly integrates features and then optimizes the integrated features to enhance the performance of a Hindi language ASR system. Initially, the sequential integration of all three feature extraction methods (taking two at a time) is evaluated. The results show that the MF-GFCC and GF-MFCC integrations outperform the MF-PLP, PLP-MFCC, GF-PLP, and PLP-GFCC integrations. Further, these feature vector integrations are optimized using three different optimization methods, PSO, C-PSO, and Q-PSO. The results describe that the Q-PSO-optimized GF-MFCC integration show significant improvement over all other optimized combinations. The developed ASR system uses HMM Tool Kit [HTK, Cambridge University Engineering Department (CUED), Cambridge, UK] 3.5 beta-2 version and MATLAB (MathWorks, MA, USA) version R2015a for its implementation.

The paper is organized as follows: Section 2 discusses the basics of feature extraction and optimization methods used in the implementation of the proposed work. The details of the proposed architecture and speech corpus are described in Sections 3 and 4, respectively. The experimental results and conclusions derived from the work are discussed in Sections 5 and 6 of the research paper.

2.1 Feature Extraction

The objective of feature extraction is to detect a set of variables from the speech signal that are correlated acoustically. Such variables are termed as features. Feature extraction has utmost importance in the development of the speech recognition system. It removes unwanted and redundant information. There are various feature extraction techniques, and this section gives theoretical background of the feature extraction methods used in the proposed work.

2.2 Optimization Methods

The optimization methods PSO, C-PSO, and Q-PSO are defined as population-based search techniques. The parameters used by a PSO algorithm are the initial size of the feature population and the fitness computation. However, C-PSO and Q-PSO use one more parameter crossover to obtain the optimal solution. The following sub-sections discuss all these methods in brief.

2.2.1 PSO

PSO is an intelligence optimization technique that belongs to a class of optimization algorithms known as meta-heuristics. It is inspired by the social behavior of animals like fish and birds. It is a very simple but powerful optimization algorithm. It is a population-based optimization technique, in which each member of the population is termed as a particle. Each particle has two components: its velocity and its current position. Every particle of the swarm communicates with each other to share information about these two components [22, 33]. In the run time of the algorithm, each particle updates its components and the algorithm finds a new particle with better velocity and position. The updated particle has better velocity and position value because it uses previous particle experiences. At the same time, each particle has its personal best experience, known as local best (Lbest_j), and a global best experience that is common for each particle, known as global best (Gbest). A particle with velocity V_j(t) and with position S_j(t) moves in the direction of vector V_j(t) and has one component parallel to Lbest_j and another parallel to Gbest. The resultant of these vectors is the new updated position S_j(t + 1) and the updated velocity is V_j(t + 1), as stated in Figure 1.

Figure 1:

Particle Swarm Optimization.

The new updated position S_j(t + 1) and velocity V_j(t + 1) of a particle are mathematically expressed as follows:

(4) Sj(t+1)=Sj(t)+Vj(t+1),

(5) Vj(t+1)=wVj(t)+C1R1(Lbestj(t)−Sj(t))+C2R2(Gbest(t)−Sj(t)),

where j is the index number of the particle; t is the time step; w is the inertia weight; R1, R2 are the uniformly distributed random variables; and C1, C2 are the acceleration coefficients. The above equations are for one-dimensional search space. The new velocity and position can also be derived for multiple dimensions, as follows:

(6) Vjk(t+1)=wVjk(t)+R1C1(Lbestjk(t)−Sjk(t))+R2C2(Gbest(t)−Sjk(t)),

(7) Sjk(t+1)=Sjk(t)+Vjk(t+1),

where j is the index number, k is the k^th component of velocity vector, and w is the inertia coefficient.

3.1 Pre-processing and Integrated Feature Extraction

The pre-processing step is used to increase the efficiency of the ASR system. Generally, the pre-processing steps comprise sampling, windowing, and denoising. Noise may arise due to microphone or electrical issues and environmental issues. Various filtering methods (e.g. Weiner filtering) are used in the pre-processing stage [1, 8]. Subsequently, the input speech signal is parameterized using various feature extraction techniques like MFCC, PLP, and GFCC, etc. Various other methods have also been developed later to increase the efficiency of the system.

The proposed work also exploits the sequential integration of MFCC, PLP, and GFCC. An important part of this sequential integration is to reduce the integrated feature space. The reduced feature space should contain only the features that contain the richest possible discriminant information. The simplest approach for feature reduction is principal component analysis (PCA) [12]. However, it has some problems that can be easily be addressed by using supervised approaches known as LDA and heteroscedastic LDA (HLDA) [17]. The HLDA technique performs better than the PCA and LDA methods. The proposed work also uses HLDA to reduce the integrated feature set. All six types of feature integrations are optimized separately for each experiment. The processing time of the proposed ASR system can be related to the time taken by integrated feature extraction methods. Full covariance matrix statistics for each component are required to estimate an HLDA transform, and the complexity of HMM decoding is directly proportional to the size of the feature vectors. Hence, both the performance and the speed of current speech recognizers by applying this feature reduction technique are improved. However, the cost of feature integration and reduction is added. Extracting features from the vocalizations begins by segmenting the waveform into frames. The frames are then parameterized into speech vectors. The time complexity for the proposed feature extraction method is O (n2V), where n is the number sequences and V is the number of frames.

3.1.1 MFCC and PLP Integrated Features

An integrated MF-PLP feature vector consisting of 17 features is obtained by combining the top 13 best features of MFCC and the top 4 best features of PLP. First-order and second-order derivatives of this sequential combination of 17 features result in a total of 51 features. HLDA is used to reduce these 51 features to 39 features. These features are termed as MF-PLP features. Figure 3 shows the steps followed to compute MF-PLP and feature extraction methods.

Figure 3:

MF-PLP Feature Extraction Method.

Similarly, all 13 features of PLP and the top 4 coefficients of the MFCC are combined and then reduced using HLDA to get the feature vector named PLP-MFCC. Figure 4 shows the steps followed to compute PLP-MFCC and feature extraction methods.

Figure 4:

PLP-MFCC Feature Extraction Method.

3.1.2 MFCC and GFCC Integrated Features

An integrated GF-MFCC feature vector consisting of 17 features is obtained by combining the top 12 best features of GFCC and top 5 best features of MFCC. First-order and second-order derivatives of this sequential combination of 17 features result in a total of 51 features. HLDA is used to reduce these 51 features to 39 features. These features are termed as GF-MFCC features. Figure 5 shows the steps followed to compute GF-MFCC and feature extraction methods.

Figure 5:

GF-MFCC Feature Extraction Method.

Similarly, all the 13 features of MFCC and the top 4 coefficients of GFCC are combined and then reduced using HLDA to get the feature vector named MF-GFCC. Figure 6 shows the steps followed to compute MF-GFCC and feature extraction methods.

Figure 6:

MF-GFCC Feature Extraction Method.

3.1.3 PLP and GFCC Integrated Features

For further considerations, GFCC is combined with PLP to combine features and obtain a feature set. An integrated GF-PLP feature vector consisting of 17 features is obtained by combining the top 12 best features of GFCC and the top 5 best features of PLP. First-order and second-order derivatives of this sequential combination of 17 features result in a total of 51 features. HLDA is used to reduce these 51 features to 39 features. These features are termed as GF-PLP features. Figure 7 shows the steps followed to compute GF-PLP and feature extraction methods.

Figure 7:

GF-PLP Feature Extraction Method.

Similarly, all 13 features of PLP and the top 4 coefficients of GFCC are combined and then reduced using HLDA to get the feature vector named PLP-GFCC. Figure 8 shows the steps followed to compute PLP-GFCC and feature extraction methods.

Figure 8:

PLP-GFCC Feature Extraction Method.

3.2 Integrated Feature Optimization

The integrated feature vectors are optimized using PSO, C-PSO, and Q-PSO methods. The PSO implementation has been done using the algorithm proposed by the authors in Ref. [11]. Like PSO, C-PSO also computes the fitness of each feature vector in the population and assigns an individual fitness value to each feature. In addition, from the population of feature vectors, the one having the best fitness is found and the global best PSO_gbest is assigned. However, unlike PSO, it uses a crossover operator. The two parents, pso_parent1, pso_parent2, with the best fitness value and a random number, pso_pfraction, in the range [0, 1] are passed. The crossover operator is then applied on these two parents to generate two offspring. Finally, the value of features is updated and the global best PSO_gbest and local best PSO_pbest values are computed iteratively. Algorithm 1 shows the application of C-PSO to obtain refined features.

The algorithm for Q-PSO-based feature optimization uses feature vector f_t as search space, qpso_{swarm_size} as swarm size, and finds two particles, qpso_parent1 and qpso_parent2, with best fitness values. A particle F_min with worst fitness value is also found. Values of Q-PSO_pbest and Q-PSO_gbest are initialized. These three particles are used in Eq. (10) to find the new particle F_new. The worst particle is replaced with the new particle, and this procedure is repeated for the whole population. The non-linear quadratic crossover operator in this proposed system finds better solution in the features search space. Algorithm 2 shows the application of Q-PSO to obtain refined features.

3.3 Acoustic Modeling

In acoustic modeling of the system, the optimized features are directly linked with the expected phones of the sentence. The two major approaches in acoustic modeling are HMM and GMM. HMM is a technique developed to deal with the statistical variations of speech. It was developed by Baum and Petrie, and is widely used in speech recognition. HMMs are stochastic finite state machines with stochastic output process attached to each state. These states can be matched by vector quantization or GMM mixture model. A uniform amount of mixtures is coincided with HMM states [24]. For a given observation sequence, the state sequence is not observable and therefore hidden. This is why the word hidden is placed before Markov models. An HMM is defined as λ (A, B, π) and consists of the following elements:

1. S_hmm: Set of states S=S1,S2,………….,Sn.

2. M_hmm: Number of distinct observation symbols per states, where individual symbols are denoted by V={v1,v2,……..,vk}.

3. A: a_ij: State transition probability, where each a_ij represents the probability of transitioning from state S_i to S_j, i.e. a_ij = P(T_t+1 = S_j|T_t = S_i).

4. B: b_j(k): Probability distribution of each symbol of hmm state, i.e. bj(k) = P(v_kat t|T_t = S_t).

5. 𝝅: Initial state distribution, i.e. the probability that S_i is an initial state.

A GMM is a probability density function represented by the weighted sum of Gaussian density components. The GMMs are commonly used in speaker recognition application because a probabilistic model for multivariate density is suitable for representation of the arbitrary densities. In the last few years, GMMs have become the powerful approach in the field of speech recognition. GMM represents a continuum of possibilities by putting a prior on some parameter describing the mean [24]. GMM can be represented as a weighted sum of K components’ Gaussian density using Eq. (11):

where X is the feature vector of D dimension and P_k(X_n), where k=1,2,…,K is the weighted linear combination of Gaussian densities. Their weighing factors, w_k, satisfy w_k > 0 and ∑k=1Kwk=1, mean vector μ_k, and the variance covariance matrix ∑k of the k^th mixture component. Each component density is represented by the Gaussian function, as follows:

The full covariance matrix can be reduced to a diagonal covariance matrix due to its computational overhead. Hence, using Eqs. (11) and (12), GMM can be defined as

where Z_k is a constant for each Gaussian.

In order to compute efficiently and to avoid underflow, probabilities are computed in log domain. Therefore, the log likelihood can be expressed using Eq. (14):

There are various advantages of the GMM model, as follows: it is computationally inexpensive, insensitive to the temporal aspects of speech, models only the underlying distribution of acoustic observation, etc.

3.4 Decoder

After the training of a system is achieved with the help of acoustic and lexicon modeling, the decoding of output variance matrix is the major objective. This objective is achieved by the implementation of the Viterbi algorithm. The search space in Viterbi at the back end can be termed as the directed acyclic graph. This word graph helps in producing the required output for each spoken utterance. The best sequence of states can be calculated with the help of previous states and observation sequence. Pattern matching is done by contemplating about the knowledge of trained model and well-defined set of phonemes. The Viterbi algorithm applies dynamic programming to determine the most suitable sequence of the HMM model. In Viterbi, to find the single best state sequence Q={q1,q2,q3,…,qt} for a given observation sequence O={o1,o2,o3,…,ot}, the quantity is defined as

where δ_t(i) is the best score along a single path, which accounts for the first t observations and ends in state S_i. The Viterbi algorithm has very similar properties to the standard forward algorithm. The major difference between them lies on the maximization of the previous state.

5.1 Performance Analysis of Various Integrated Feature Vectors

The performance comparison of the implemented ASR system for all feature extraction integrations including the conventional method MFCC at the front end and HMM-GMM acoustic modeling at the back end is shown in Table 1. It can be clearly concluded from the comparison that MF-GFCC- and GF-MFCC-based ASR systems have more recognition accuracy than the other integrated feature vector-based ASR systems. Also, while testing Set3 with training Set3, where both datasets include north and south Indian male and female speech utterances, the ASR systems perform better than all other combinations. Also, the comparison of the proposed feature integration methods has been shown with traditional MFCC features [11]. The GF-MFCC combination has its highest WRr at 11.15%, which is better than the highest WRr of MFCC. One more interesting observation is that the accuracy rate of each training dataset with respect to another training dataset shows mixed behavior. For example, the training dataset Set2 and testing dataset Set2 combination has more accuracy than the training dataset Set1 and testing dataset Set2 combination in the MFCC, MF-PLP, MF-GFCC, and GF-MFCC integrations. However, the same combination has less accuracy in the PLP-MFCC, PLP-GFCC, and GFCC-PLP integrations. Similar distinct observations can also be made in training dataset Set3. Hence, different variabilities like speaker variability, speech variability, etc., play a major role during training and testing of an efficient speech recognition system.

5.2 Performance Analysis Using Different SNRs

A real-time speech recognition system is required to be robust to real-time noise interference to the speech input signal. A robust system is categorized into two major types: one that alters itself according to any type of noise based on the adaptation technique and another that decreases the noise interference from noisy audio inputs based on reduction techniques [10]. The proposed system is evaluated using additive noise with different SNRs. The interference added to the input speech signal while processing through a communication channel is known as additive noise. The proposed speech recognition system uses additive white Gaussian noise with the help of the NOISEX92 database [32]. It is the type of additive noise that has uniform power throughout the frequency band, and it has uniform normal distribution in the time domain. The accuracy values of the implemented ASR system using MF-GFCC and GF-MFCC feature vector combinations with different SNRs are shown in Table 2. The results clearly show that the GF-MFCC-based ASR system performs better than the MF-GFCC-based ASR system, and the accuracy rate increases with increasing SNR value.

5.3 Performance Analysis Using Refined and Integrated Feature Vectors

The integrated features MF-GFCC and GF-MFCC vectors are refined using the optimization methods PSO, C-PSO, and Q-PSO. The accuracy of the implemented ASR system using these two refined features and HMM-GMM-based acoustic models is tested against all three test datasets. The results in Table 3 show that the GF-MFCC-based system performs better than the MF-GFCC system in all classifications. Also, it can be concluded from Table 3 results that Q-PSO-optimized integrated feature vectors outperform C-PSO- and PSO-optimized feature vectors. The accuracy increase of the Q-PSO-optimized ASR system over PSO- and C-PSO-optimized ASR systems with different dataset combinations lies in the approximate range of 1%–3%.

5.4 Performance Analysis Based on Different Speaker Modes

On the basis of the speaker mode, ASR systems can be classified in to three categories: speaker-dependent (SD), speaker-independent (SI), and speaker-adaptive (SA) ASR systems. To perform accurately, prior training of the individual speakers’ voice is required by SD ASR systems, whereas SI ASR systems do not require such training and SA ASR systems try to adapt its operation to the characteristics of new speakers. The performance of the developed systems is tested using all these three types of speaker dataset in clean as well as noisy conditions. For SA classifications, the maximum likelihood linear regression adaptation approach is used. Table 4 describes the results for all possible combinations. It can be observed from the results that SD ASR systems have more WRr (%) than SI and SA ASR systems for all dataset combinations. The SA ASR system outperforms SI ASR systems for all Q-PSO-optimized dataset combinations. However, for dataset combinations in C-PSO, SI ASR systems outperform SA ASR systems and vice versa.