Mental stress is known as one of the main influential factors in development of different diseases including heart attack and stroke. Thus, quantification of stress level can be very important in preventing many diseases and in human health. The prefrontal cortex is involved in body regulation in response to stress. In this research, functional near infrared spectroscopy (fNIRS) signals were recorded from FP2 position in the international electroencephalographic 10–20 system during a stressful mental arithmetic task to be calculated within a limited period of time. After extracting the brain’s hemodynamic response from fNIRS signal, different linear and nonlinear features were extracted from the signal which are then used for stress levels classification both individually and in combination. In this study, the maximum accuracy of 88.72% was achieved in classification between high and low stress levels, and 96.92% was obtained for the stress and rest states. Our results showed that using the proposed linear and nonlinear features it is possible to effectively classify stress levels from fNIRS signals recorded from only one site in the prefrontal cortex. Comparing to other methods, it is shown that the proposed algorithm outperforms other previously reported methods using the nonlinear features extracted from the fNIRS signal. These results clearly show the potential of fNIRS signal as a useful tool for early diagnosis and quantify stress.