Introduction

Good eating habits play a crucial role in the prevention and management of Food-related Health Conditions (FrHCs) such as obesity and malnutrition, and non-communicable chronic diseases (NCDs) such as diabetes and hypertension1. Consequentially, from traditional dietary assessment methods such as manual logging or semi-automated logging via smartphone apps and cameras to automated monitors including smart objects (e.g., smart tray2, utensils3, etc.) and wearables (e.g., smart collar4, smart earbud5, etc.), the technology surrounding Automatic Dietary Monitoring (ADM) has always been a focus of ubiquitous computing in search of solutions that is convenient, unobtrusive and user-friendly for sustained usage, to help people cultivate a healthy dietary habit. In addition, ADM can also provide valuable information for remote health monitoring systems.

Existing ADM techniques can be grouped into vision-based and sensor-based methods. Vision-based methods usually require capturing images of the food or the person to extract the features, which often achieve high accuracy given high-quality images for hundreds of food categorization tasks with deep learning models6,7,8,9,10. However, for sustained usage in the real world, vision-based methods are often influenced by image quality (e.g., lighting conditions, stability), usability (e.g., requiring users to photograph their food), and private issues (esp. for automatic photo-taking methods), which can be better addressed by sensor-based approaches. Smart sensing in ADM can be used as smart objects, which monitor interactions such as handling and feeding food2,3,11,12,13,14. For smart wearables, sensors are generally used to infer dietary activities from dietary-related gestures15,16, or from characteristic physiological signals propagated through the body during food ingestion (e.g. swallowing and chewing)12,17,18. Compared to wearables, smart utensils are more directly in contact with the food and are less prone to activities irrelevant to the dietary context. However, for ADM specifically, wearable devices have unique usability advantages over smart objects, as the system can accompany the user unattended. For example, the users do not need to carry a set of smart utensils for every meal and maintain the hygiene of the dedicated utensils.

Automatic dietary monitoring

Many sensor-equipped smart objects and wearables for ADM have emerged in recent years. For example, a smart fork integrated with an IMU sensor was proposed to detect food pick-up gestures and estimate the amount of food in3. Zhou et al.2 proposed a smart dining tray based on pressure sensors, which can classify eight kinds of food intake-related activities, such as stir, scoop, and cut, with an accuracy of 94.6%. In addition, smart spoons using light spectrum13, smart bottles using an ultrasonic sensor19, and smart cups using an impedance sensor20, were studied by many researchers for dietary monitoring. Apart from smart objects, wearable devices for ADM are also widely studied21,22,23,24, which enjoy unique advantages over smart objects, like portability and convenience. For example, Cheng et al.4 designed a textile neckband with four capacitive sensors for activity recognition and ADM in everyday situations. Yin et al.12 presented the AutoDietary, with a high-fidelity neck-worn microphone, to monitor and recognize food intake of seven kinds of food, including fluid and solid food, with an accuracy of 84.9%. In general, these sensors, like IMU, acoustic, capacitive, optical, FSR, and gas sensors, are the most frequently used in the existing works for ADM.

Bioimpedance sensing

Impedance sensing is widely used in the area of biological analysis and food characterization25. The impedance sensor has been integrated into smart utensils20,26 to extract information on food intake. In wearables, impedance sensing is used primarily to monitor human physiological status, such as breathing activity27, blood pressure28, cardiography29 and body composition30, where users are often required to remain steady during data acquisition, and the signal variation due to motion and interaction with objects is often discarded as artifacts31 in existing works. However, we argue that such signals are not a result of meaningless noise but rather activities that could be exploited especially for ADM applications, thus providing an opportunity for an atypical use of bio-impedance wearables for ADM. While some similar works about bio-impedance sensing-based activity recognition have been proposed recently, like fitness activity32 and hand-over-face gesture recognition33, the application scenarios in ADM based on bio-impedance-sensing has not been well studied.

In this work, we present iEat, an unobtrusive wearable system that leverages the electrical conductivity of the human body (e.g., hands and mouth), normal metal utensils, and food pieces to detect dietary activities through impedance signals measured from circuit loops consisting of these conductive components.

For example, actions involving food separation create a dynamic circuit through the food piece; the moments of ingestion are also associated with a unique circuit bridge from one hand to the mouth. Either the use of bare hands or normal metal utensils will form the loops of those circuits. To the best of our knowledge, our work is the first to leverage a bio-impedance wearable device to recognize both food intake activity and types of food by data mining through what are usually discarded artifact signals in classical bio-impedance sensing applications.

Overall, we have the following two contributions from this work:

  1. 1.

    With iEat, we proposed a novel wearable ADM paradigm with impedance sensing across two wrists through an abstracted dynamic human-food interaction circuit model.

  2. 2.

    We evaluated iEat with a realistic dining experiment of 40 meals performed by ten subjects and demonstrated its reliability in dietary activity recognition and food type classification with an average macro F1 score of 86.4% and 64.2%, respectively.

  3. 3.

    We further discussed the potential of using iEat to count the number of food items consumed. An average counting error rate of 11.48% was achieved in the experiment.

Note that the sensing principle in this paper is not based on the absolute impedance value but on the impedance signal variation caused by dietary activities. For this reason, the more precise four-electrode impedance measurement configuration (removing the effect of the measured electrodes) was not selected, a simpler two-electrode configuration was implemented in this work.

Materials and methods

General principle

iEat is built upon the fact that both the human body and food are conductive objects, which can be represented as electrical components in a circuit model25. Figure 1 shows the schematic diagram of the iEat; the abstracted human-food impedance model includes two branches: the body circuit branch ((El-–Zal–Zb–Zar–Er) and the food circuit branch (El–Zf–Er), which are joined in parallel when two electrodes (Er, El) are attached to the wrists during the cutting of food using the fork and knife. The measured impedance consists of body impedance (Zb), arm impedance (Zar, Zal), food impedance (Zf), and contact impedance such as Zt and Zs shown in Fig. 2. The human-food circuit model is a dynamic circuit that changes with the body’s motions and interactions with metal utensils and food during dining activities. For example, during food ingestion using a hand or a fork, the food circuit branch becomes disconnected, and a new branch is formed from one wrist through the mouth to another wrist. While cutting food, for example, the food branch is repeatedly switching between open and closed. Such alterations can lead to a notable and immediate change in impedance measurement results. Moreover, the impedance of the food circuit branch is mainly influenced by the electrical properties of the food. Consequently, monitoring and analyzing the fluctuation patterns in the overall impedance of the proposed abstracted human-food circuit model makes it possible to automatically track information regarding food intake-related activities and the types of food consumed.

In the food intake-related activity classification task, four types of activity (cutting, drinking, eating with a hand, and eating with a fork) are recognized in this study, as shown in Fig. 2. Any other activities where the two hands are positioned separately are grouped into the IDLE status (null class). According to the abstracted circuit model, the impedance value could theoretically be calculated as Eq. (1). The \(Z_{s1}\) and \(Z_{S2}\) are the skin impedance between the electrode and the internal body. The skin acts like an electrical component such as a capacitor, allowing more current to flow if a voltage changes rapidly34. When cutting food, the entire impedance of the human-food model can be calculated by Eq. (2), and the contact impedance \(Z_t\) comes from the contact between utensils, hands, and food. Since the new parallel circuit branch is generated, the impedance should be smaller than the IDLE status. When eating with a fork or drinking with straw, a subcircuit branch from the hand to the mouth via a metal fork or straw is generated, \(Z_{b1}\) is part of \(Z_b\), Eq. (3) shows the computation of the total impedance, where the impedance of the utensils and the impedance of contact between the utensils and the hand and the mouth are ignored. Thus, the new branch shorts out part of the body and arm impedance circuit path. When eating by hand, the current flows into the mouth through the food if the hand does not contact the mouth directly. The whole impedance can be calculated by Eq. (4) when ignoring the contact impedance among food, hand, and mouth.

$$\begin{aligned} Z_{idle}= & {} Z_{s1} + Z_{al} + Z_{b} + Z_{ar} + Z_{s2} \end{aligned}$$
(1)
$$\begin{aligned} Z_{cutting}= & {} \frac{(Z_{s1} + Z_{al} + Z_{b} + Z_{ar} + Z_{s2}) * (Z_f+Z_t)}{(Z_{s1} + Z_{al} + Z_{b} + Z_{ar} + Z_{s2})+ (Z_f+Z_t)} \end{aligned}$$
(2)
$$\begin{aligned} Z_{eating(fork)}= & {} Z_{b1} + Z_{al} + Z_{s1} \end{aligned}$$
(3)
$$\begin{aligned} Z_{eating(hand)}= & {} Z_{b2} + Z_{ar} + Z_{s2} + \frac{(Z_{s1}+Z_{al}+Z_{b1})*Z_f}{(Z_{s1}+Z_{al}+Z_{b1})+Z_f} \end{aligned}$$
(4)
Figure 1
figure 1

Schematic diagram of iEat. (a) the abstracted human–food interaction model (Zal: left arm impedance, Zb: upper body impedance, Zar: right arm impedance, Zf: food impedance, El: electrode on left wrist, Er: electrode on right wrist). When the human cuts the food, two circuit branches are built up, including all conductive components (El–Zal–Zb–Zar–Zr; El–Zf–Er). Otherwise, the branch (El–Zf–Er) remains open. (b) the hardware architecture includes an analog front-end based on AD5941, IMU sensor, microcontroller, and BLE interface. The measured result is transmitted to the laptop via BLE. (c) the hardware prototype with a dimension of 50 × 32 × 11 mm, (d) the raw signal with the denoted eating episode. The variation in signal amplitude of cutting is the biggest, while eating by hand is the smallest. Eating by fork is usually performed after cutting food.

Full size image
Figure 2
figure 2

The abstracted human-food interaction circuit model, and the unique signals of different activities. These activities can be classified using different neural network models that receive the input sliding windows generated from the raw signal and extract features from raw signal automatically.

Full size image

Hardware design

The iEat wearable hardware prototype shown in Fig. 1 was designed to monitor the impedance signal of the human-food circuit model that occurs as users handle food like cutting, eating food, and drinking beverages, allowing for precise tracking of food intake activity and classify the types of food consumed. The iEat prototype consists of four modules: the analog front-end (AFE) module, the control module, the electrodes, and the power supply. The chip AD5941 (Analog Devices) was selected as the core component of the analog front-end module of iEat, as it can both generate the voltage stimuli as sinuous signals with a configurable frequency from 0.015 to 200 kHz and measure the response current signal with an integrated high-speed trans-impedance amplifier. It also has an integrated FFT hardware accelerator to decipher the real and imaginary components from the measurement. An nRF52840 (Nordic®) is the controller module that drives the AFE through an SPI bus and transmits the measurement result to the computer via Bluetooth. The wet Ag/AgCI electrodes interface the user’s body with the AFE. The work current of iEat is around 20 mA. A compact lithium battery with a 500 mAh capacity powers the system, which could power the system for around one day.

Study population

In this study, ten volunteers (four females and six males) from seven countries of origin were invited to have four meals wearing the iEat prototype on different days. Table 1 shows the overview of the characteristics of the study population, including the height, weight, arm length, chest size, and BMI of the population. The height and weight range among the participants from 160 to 188 cm and 53 to 114 kg, respectively. The Body Mass Index (BMI) ranges from 17.2 to 39.4. The participants signed an informed consent form before the study.

Table 1 Overview of the study population characteristics.
Full size table

Experimental protocol

In this study, the impedance between the body and food was obtained by generating current stimuli injected into the body and measuring the response. The amount of current injected into the body should conform to IEC 60601 standards, which limits the amount of DC (direct current) and AC (alternative current) current entering the human body. The amount of direct current should be less than 10 µA, and the maximum allowable AC current is 500 µA at 50 kHz and 600 µA at 60 kHz35. The internal body resistance is about 300 Ohms, related to the wet, relatively salty tissues beneath the skin34. In addition, skin resistance will be added when measuring body impedance, which can be effectively bypassed only if the skin breaks down due to high voltage, a cut, a deep abrasion, or immersion in water. In this study, the skin to which the electrodes were attached was in good condition. Here, assume that the body impedance, including the skin impedance, is 300 Ohms. To comply with the IEC 60601 standards, the maximum output voltage of the stimuli signal at 60 kHz should be less than 180 mV p–p (peak to peak) according to Eq. (5). In this experiment, a 60 kHz sinus signal without DC bias was added to the body through the wrists via the two electrodes. The output voltage of the stimuli is 50 mV peak to peak to guarantee a safe amount of current entering the body. The AFE output data rate is configured as 20 Hz.

$$\begin{aligned} V_{stimuli} = 300 \, {\text{Ohms}} * 0.6 \, {\text{mA}} = 180 \, {\text{mV}} \end{aligned}$$
(5)

Participants were asked to sit in front of the table and eat the targetted food during each meal in a daily living environment, with an average meal duration of approximately forty minutes. They were allowed to socialize with others during the meal. The measured impedance data was transmitted to a web application on the laptop via Bluetooth. The web application was written in JavaScript, which was used to observe the impedance signal in real time and store the data in text files on the laptop. Also, there was a camera in front of the participants to record the video used to label the data after the experiment. The TRAINSET tool from Geocene was used to label the time series data. The food was consumed using different conventional utensils; for example, volunteers used a metal fork to eat meat and vegetables, used a metal straw and metal cup to drink beverages, and ate fruits and bread directly by hand. They were asked to use a metal knife and fork to slice certain foods listed in Table 2.

Table 2 Food information.
Full size table

Raw data and preprocessing

Figure 3 shows an example experimental session and sensor data collected from subject B; the whole session lasted around one hour. The raw signals consist of both real and imaginary components obtained directly from the hardware Fast Fourier Transform module of the analog front-end chip, from which the magnitude and phase signal could be derived. The raw data is marked by five activity labels, including the Null class. The eating, cutting, and drinking episodes were performed alternately during the experiment. While the one-session raw data was collected from the same subject, it’s important to note that there is a noticeable shift in the amplitude of impedance data throughout the entire session. The magnitude and real component have shifted towards smaller values, whereas the phase and imaginary component values have drifted in the opposite direction as shown in Fig. 3. Furthermore, it’s worth noting that the placement of electrodes on the wrist can significantly influence measurement outcomes, potentially resulting in impedance variations observed between different sessions involving the same subject. Moreover, the absolute impedance signal is intricately linked to bodily factors such as water and fat content. Participants with varying heights and weights exhibit a broad range of body impedance values (Fig. 4).

Figure 3
figure 3

Example experimental session and sensor data collected from subject B.

Full size image
Figure 4
figure 4

Boxplots representing the distribution of the impedance signal between different subjects without conducting any food-related activity.

Full size image

This Boxplot shown in Fig. 4 illustrates the distribution of impedance data, encompassing magnitude, phase, real, and imaginary component values, collected from different subjects when placing both hands separately without engaging in any food-related activities. Each ”box” represents data computed from eight sampled impedance measurements, with two readings taken at the beginning and end of each session. Each subject underwent four sessions. Among the ten subjects, the magnitude values ranged from 600 to 1600 Ohms. Notably, subject C exhibited the most significant variation in magnitude across the four sessions, with a deviation of approximately 600 Ohms. This variation was attributed to a loose electrode-skin contact in one session. The remaining subjects experienced magnitude variations of roughly 100 Ohms or less throughout the entirety of the experiment. In addition, the distribution of impedance value of the users performing food related activities is shown in Fig. 5.

Figure 5
figure 5

Boxplots representing the distribution of the impedance signal conducting different food-related activities.

Full size image

Since the absolute impedance value among the ten participants demonstrates different amplitude and the impedance variation information caused by food-related activities plays an essential role in the classification task by iEat, it is imperative to apply a preprocessing method that aligns the distribution of signal variations resulting from eating activities. Therefore, to address this issue, a session-wise standardization procedure was implemented on the raw data. As food intake-related activities vary in duration, tasks such as cutting food typically require more time compared to the act of eating with one’s hands or a fork. Besides, the eating speed between different subjects is also different. In this study, the classification performance with different window sizes of the instances, including the length from 10 to 60 with step of 10 was investigated. The overlap length of each instance is 10. The label of each instance was determined by majority voting inside the window.

Figure 6 shows the t-SNE plots of the four activity’s feature distribution with different window sizes. The input instances encompass magnitude, phase, real, and imaginary components. Our approach involved initially employing Principal Component Analysis (PCA) to extract the 20 most important components from the raw instances. Subsequently, we utilized t-SNE to further reduce these 20 components into the two most significant ones. Both the PCA and TSNE functions come from the Python package scikit-learn. The t-SNE plots reveal that as the window size increases, the boundaries between the activities become more distinct. When using the smallest window size of 10, most extracted features from the four activities tend to intermingle as shown in Fig. 6a, which is difficult for segmentation. Therefore, a window size of 10 will not be considered an input to the neural network. The performance of the rest window sizes for the activity and food classification was evaluated.

Figure 6
figure 6

t-SNE plots of the four activity’s feature distribution with different window size.

Full size image

Evaluation method

To evaluate the performance of the iEat for Automatic Dietary Monitoring including four kinds of food intake-related activity recognition (cutting, drinking, eating (hand) and eating (fork)) and seven kinds of food classification (steak, sausage, cucumber, tomato, mineral water, vitamin water, and banana) across the subjects and meals, two leave-one-out cross-validation methods were applied in this study, such as Leave-one-person-out and Leave-one-meal-out, as Fig. 7 shows. Furthermore, our study delved into the performance of various input features for the task, encompassing four distinct types of input features derived from different combinations of impedance data. These combinations included single-channel magnitude, magnitude and phase, real and imaginary components, as well as the use of all four features in unison. Additionally, we assessed the impact of different window sizes on the classification task’s performance. We employed four commonly used models to process the impedance data and evaluate the effectiveness of impedance sensing for activity recognition. These models included convolutional, recurrent neural network (RNN), and transformer layers. Details regarding these models are provided below:

  1. 1.

    DeepConvLSTM36 it was designed for human activity recognition based on time series data and often served as a baseline in many research works. In this study, the DeepConvLSTM network has four convolution layers, followed by two LSTM layers and one linear layer. The kernel size is 3, and each convolutional and LSTM layer has 256 filters. The output channel of the linear layer is equal to the class number of the activity.

  2. 2.

    TinyHAR37 The TinyHAR was also designed specifically for human activity recognition including convolutional, transformer, LSTM, and linear layers. In this study, TinyHAR was employed for the classification task. The kernel size of the convolutional subnet from the TinyHAR was configured as 3 with 20 filters in each convolutional layer. The TinyHAR has the smallest model size among the four models.

  3. 3.

    TCNNet The TCNNet was built upon the existing work38. This architecture was selected as it was successful in many sequence modeling tasks, outperforming even LSTMs39. In this study, the TCNNet has four TCN blocks with 128 filters among each Convolutional layer.

  4. 4.

    ConvDense The convDense Network was similar to the architecture of the DeepConvLSTM, but we used the dense layers to replace the LSTM. In this study, there were three convolutional layers with the filter numbers 400, 200, and 100, correspondingly.

All models were implemented using PyTorch and trained for 200 epochs, with an early stopping criterion set to patience 30. The summation of Accuracy, macro F1 score, and weighted F1 score served as the stopping metric. The training process utilized the cross-entropy loss function and the Adam optimizer, with a learning rate of 0.01 and a batch size of 128.

The F1-score40, a composite metric that considers both precision and recall, serves as a performance benchmark for evaluating the effectiveness of iEat in activity classification and food recognition. This metric holds particular significance due to its resilience in the face of class imbalance. This scenario arises when certain activities have a greater number of instances compared to others. In this study, the number of cutting instances in the activity classification task is much more than in the other three kinds of activities. In particular, there is a set of different average F1 scores, such as macro, weighted, and micro. This study selected the macro F1 score as the metric, computed using the arithmetic mean (unweighted mean) of all the per-class F1 scores. Thus, it treats all classes equally.

Figure 7
figure 7

Leave-one-out cross validation.

Full size image

Results

Activity classification result

Table 3 shows the summarized result of activity classification, including the combinations between five different window sizes, four different models, and four different features. The best activity classification results, as obtained from the leave-one-person-out and leave-one-meal-out validation schemes, averaged macro F1 scores at 86.2% and 86.4%, respectively. Impressively, these outstanding outcomes were consistently achieved through utilizing the TinyHAR model, employing a window size of 30 and input features comprising both Magnitude and Phase components. Figure 8 shows the point diagram of the average macro F1 score of the activity recognition with different window sizes, neural models, and features. Within the leave-one-person-out validation scheme, the performance of the models showcases a clear trend of degradation as the window size increases. However, it is noteworthy that the TinyHAR model stands out as an exception to this trend, consistently delivering its best performance when configured with a window size of 30 as shown in Fig. 8a. Among the models tested, those utilizing input features such as a single channel from magnitude or two channels from real and imaginary components consistently exhibited comparatively lower performance across various window sizes. Remarkably, the models utilizing two-channel features derived from magnitude and phase consistently outperformed all others across different window sizes. The performance surpasses even that of models using four-channel features, including magnitude, phase, real, and imaginary components. In the leave-one-meal-out validation schemes, the two-channel features derived from magnitude and phase consistently outshine other feature combinations across all models, regardless of the window size employed. Conversely, models using solely single magnitude information as input have consistently yielded the lowest macro F1 scores as shown in Fig. 8b. In general, the combination of magnitude and phase features has proven to be more informative for neural network models during the training process for activity classification. Notably, the TinyHAR model consistently outperforms other models, underscoring its superior performance. Figure 9 shows each subject’s leave-one-person-out macro F1 score from the TinyHAR model with a window size of 30 and the input features of magnitude and phase. It is worth noting that the optimal window size for achieving the best macro F1 score varies among different subjects. For instance, when using a window size of 20, subject B achieved the lowest macro F1 score at only 72%, while subject I attained the highest score at 93%. This discrepancy can be attributed to variations in the speed at which different subjects perform activities. If the window size is too small and a subject engages in a slower-paced activity, important activity features might be missed or inadequately captured. In general, when a window size of 30 (equivalent to 1.5 s) is employed, the macro F1 scores for activity classification across all subjects consistently exceed 82%.

Table 3 Summary of average micro F1-score of eating activity classification.
Full size table
Figure 8
figure 8

Average macro F1 score of food-related activity recognition with different window sizes, neural models, and features. (M Magnitude, P Phase, R Real component, I Imaginary).

Full size image
Figure 9
figure 9

Macro F1 score of each subject with different window sizes (Model TinyHAR, Feature M + P).

Full size image

Food recognition result

In the food recognition tasks, all instances related to cutting activities from the activity classification task are categorized into distinct classes based on the type of food being cut. Furthermore, instances involving drinking activities are separated into two classes based on the type of beverage consumed, considering that vitamin water typically exhibits different conductivity from natural water. The food impedance is added to the human-food circuit model when users employ a fork and knife to interact with their food during the cutting phase. This impedance data analysis enables the recognition of various food types. Table 4 illustrates the food classification results across seven distinct categories, encompassing various validation schemes, input features, models, and window sizes. It is noteworthy that the highest macro F1 scores, which stand at 62.7% and 64.2%, were achieved by the TinyHAR model. These top-performing results were obtained when employing input features derived from magnitude and phase data, coupled with a window size of 60, for both the leave-one-person-out and leave-one-meal-out validation procedures. In this particular task, it’s evident that the performance of features extracted from single-channel magnitude consistently lags behind other feature combinations across all models utilizing various input features as shown in Fig. 10. The limitation of single-channel magnitude features becomes apparent due to their inability to capture the complete information regarding the capacitance and inductance in the circuit models. Modeling food as a pure resistor in the circuit model is not feasible. Input features derived from both magnitude and phase consistently exhibit superior performance across all models, with the exception of the DeepConvLSTM model. In the case of the DeepConvLSTM model, the feature combination involving magnitude, phase, real, and imaginary components slightly outperforms the model using only magnitude and phase features in terms of macro F1 score. It’s interesting to note that, in contrast to the activity classification task, the food classification performance demonstrates a continuous improvement as the window size increases across all models with various input features and under different cross-validation schemes. Figure 11 shows the food classification macro F1-score of each subject by TinyHAR model with different window sizes. The performance of food classification shows improvement as the window size increases across all subjects. This observation holds even though each subject may have a different pace when cutting food. This contrasts with the activity classification task, where the impact of window size on performance varied among subjects.

Table 4 Summary of average micro F1-score of food classification.
Full size table
Figure 10
figure 10

Average macro F1 score of food classification with different window sizes, neural models, and features. (M Magnitude, P Phase, R Real component, I Imaginary).

Full size image
Figure 11
figure 11

Food classification macro F1 score of each subject with different window sizes (Model TinyHAR, Feature M + P).

Full size image

Discussion

In this study, we introduced the innovative wearable device iEat, which utilizes impedance sensing to classify not only food intake-related activities but also food types. This research demonstrates that valuable information about food intake activities and food types can be extracted from the impedance data collected through the proposed abstracted human-food circuit model. We assessed the performance of iEat using the macro F1 score as the evaluation metric, employing two cross-validation methods: leave-one-person-out and leave-one-meal-out.

Activity classification

Figure 12 shows all the combined confusion matrices of the activity classification from the TinyHAR model with a window size of 30 and the input features of magnitude and phase. It can be observed that the cutting class can be recognized with the highest recall of 93% in the leave-one-subject-out and the leave-one-meal-out procedures. In contrast to other activities, the food circuit branch becomes fully integrated into the body circuit when a user cuts food with a fork and knife. As a result, the overall impedance of the parallel circuit closely resembles the impedance of the circuit branch with the lower impedance as the Eq. (2) described, which leads to a significant reduction of the amplitude of the impedance data compared to the IDLE status as shown Fig. 3. While all four types of activities can induce amplitude variations in the impedance data, the measurement results indicate that the cutting activity exhibits the most significant amplitude reduction, except for the activity of putting the hands together. The signals associated with putting the hands together are recorded at the start and end of each session and are categorized under the null class as shown in Fig. 3. The substantial amplitude variation observed during the cutting activity is a noteworthy characteristic that can greatly enhance recognition recall. In the leave-one-person-out procedure, the most confusing activities are cutting and eating with a fork, as shown in Fig. 12a. The cutting activity is followed by eating with a fork closely. If the duration between cutting and eating(fork) is very short, which could happen to these subjects who have very fast eating habits, especially for cutting soft food, like bananas, then the instances with a window size of 30 (1.5 s) could include the pattern of both activities, whose label is given by the majority voting method. Such kinds of instances could confuse the model during training. This similar result is also presented in the leave-one-meal-out schemes as shown in Fig. 12b.

Figure 12
figure 12

All combined confusion matrices for activity classification.

Full size image

Food classification

The fundamental principle underpinning food classification in this study relies on the impedance information of the food as measured by iEat during the cutting process using metal utensils (knife and fork). In theory, if there exists a significant distinction in food impedance, the results of food classification could approach a high level of accuracy, akin to prior research achievements in beverage classification where noticeable differences in impedance were leveraged for classification20. However, iEat is meticulously designed as a wearable device, offering distinct advantages such as portability and convenience. Its electrodes do not make direct contact with the food. As a result, the impedance measured by iEat during the cutting activity comprises not only the food but also includes contributions from the human body and the contact impedance between electrodes, skin, and food. This multi-faceted impedance composition significantly adds to the complexity of the food classification task for iEat, making it a more challenging endeavor. In addition, it’s worth noting that during the food-cutting process, the distance between the fork and knife is not consistent and can vary. This variability in the spatial configuration can also contribute to different impedance measurements for the same types of food, further complicating the food classification task. Therefore, iEat could not distinguish these kinds of food whose impedance is very close. Figure 13 shows the combined confusion matrices of the food classification from the TinyHAR model with a window size of 60 (3 s) and the input features of magnitude and phase. It can be observed from the confusion matrices that the confusing classes of the two different cross-validation methods are the same, which can be grouped into three main types: meat, vegetables, and beverages. For example, the most confusing classes are steak and sausages, as shown in Fig. 13a. The two kinds of food are made of meat, which has closer water content and similar textures than vegetables like cucumber and tomato. Thus, the difference in the features extracted from the two kinds of food by the neural networks could be tiny, which confuses the classifier. The cucumber and tomato also can not be well recognized by iEat, whose recognized recall is around 65%. The recognized recall of the banana class is higher than the meat and vegetables because the banana is the softest food in this study, which users can cut easily and quickly; The raw impedance signal of cutting the banana looks like a triangle wave, while the signal of cutting other food looks closer to a square wave as the raw signal shown in Fig. 1. This feature provides valuable information to the neural network to recognize it. In the leave one meal out procedure, the banana can be recognized by iEat with the highest recall of 79%. The beverage was recognized when the user was drinking with a straw, which is different from the recognition of other food during cutting; the drinking and cutting activity can be well distinguished by iEat. Thus, the beverage and other food classes are almost never confused by iEat. Although nature water and vitamin water have different conductivity properties, leading to different impedance measurements, the two kinds of beverages still confused the iEat, whose recognition recall is around 70% because of the indirect contact between the electrodes and beverages. In general, iEat can classify meat, vegetables, beverages, and fruit well in this study.

Figure 13
figure 13

All combined confusion matrices for food classification.

Full size image

The potential of food ingestion counting

The number of bites taken could be a valuable indicator for estimating how much food people consume. The food ingestion activity, like eating with a fork, can cause an obvious variation of the bio-impedance signal, which looks like an inverted peak signal as shown in Fig. 2. Thus, the number of bites taken could be deduced by counting the number of peaks caused by food ingestion activity in the raw magnitude signal. The possibility of counting the repetition of eating with a fork based on the impedance signal was further discussed in this work. The other activities, like cutting, can also cause an inverted peak signal, as shown in Fig. 14, where the raw signal was flipped and shifted to apply the peak detection. Although the height of the peaks between eating and cutting activity is different, which could be distinguished by a configurable threshold value, the magnitude shift during a meal and magnitude difference among different subjects could cause many false positive predictions. Besides, the thresholds need to be set manually for different subjects. To avoid these issues, we proposed an automatic bites counting solution by combining the peak detection and neural network model trained for eating activity classification. Firstly, a session-wise standardization procedure was implemented on the raw magnitude signal, and the standardized raw magnitude signal was smoothed by a low pass filter (Butterworth filter) with a cutoff frequency of 1 Hz. Then the filtered signal was flipped to apply peak detection. The find-peak function with a prominence of 0.3 from the Python package scikit-learn was used to find the position of the peaks whose prominence is above 0.3. The model’s input window is set to a size of 30, encompassing both magnitude and phase data, which includes 15 samples preceding and following the detected peak position. Next, the trained TinyHAR model for eating activity classification was used to detect whether the peak belongs to the activity of eating with a fork.

Figure 14
figure 14

Example of peak detection in magnitude signal with the configuration of thresholds manually (the raw magnitude signal was flipped and shifted to apply the peak detection function).

Full size image

The metrics of \(Error_{rate}\) and precision were used to describe the performance of our proposed method for food ingestion counting. The \(Error_{rate}\) is defined as Eq. (6) shows according to the existing work38. Since the predicted count number contains both True Positive (TP) and False Positive (FP) predictions, the metric precision was also used for further evaluation of the performance of our proposed solution. Precision can describe the proportion of accurately predicted positive observations to the total predicted positive observations, serving as a measure of the model’s ’exactness.’, which is defined as Eq. (7).

$$\begin{aligned} Error_{rate}= & {} \frac{\mid Count_{predicted} - Count_{true}\mid }{Count_{true}} \end{aligned}$$
(6)
$$\begin{aligned} Precision= & {} \frac{TP}{TP+FP} \end{aligned}$$
(7)

To evaluate the potential of food ingestion counting based on the bio-impedance signal, we used the same dataset as the eating activity classification experiment, including ten subjects. The magnitude and phase were selected as the input features. Figure 15 shows the food ingestion counting results among ten subjects. It can be observed that the average counting rate of 11.48% was achieved by the proposed automatic eating peak counting method. It’s worth mentioning that the detection precision is above 90%, which indicates that the proportion of accurately predicted positive observations is higher than 90%. However, the food ingestion counting results among the ten subjects are different, which mainly resulted from two reasons: (1) the peak detection function with the same parameters was applied to select the input instances of the neural model, which could miss some peaks caused by food ingestion in some subjects. (2) the model used to detect the food ingestion activity based on the detected peak was trained in leave-one-subject-out way, some instances can not be correctly recognized, as the food ingestion speed and body impedance among the subjects. Besides, some peaks caused by irrelated activity with the similar prominence could also be counted. A multi-modality method, like the combination of impedance sensor and IMU sensor, will be studied in future work to improve food ingestion counting result.

Figure 15
figure 15

Boxplot of food ingestion counting result using a combination of peak detection and neural network model.

Full size image

Limitation

Although we demonstrated that iEat supplies impressive potential for wearable ADM for eating activity classification and food recognition, we observed some limitations that need to be addressed in future work. On the wearable usability aspect, while iEat can be used with any metal utensils or bare hands, direct contact between the electrodes and the user’s skin is needed for continuous impedance monitoring to obtain high-quality signals. This requirement may result in discomfort depending on the electrode selection, especially for extended periods of time. Besides that, the two electrodes on each wrist need to be wired to a single AFE. While we did not observe any interference on the wire, the wire should be integrated into garments for ease of use. Such inconvenience could be addressed by using wirelessly coupled bio-impedance solutions in future work41. Furthermore, we used fixed-frequency stimuli at 60 kHz to measure the impedance, while multiple frequency sweeps to record the impedance spectrum may reveal extensive information about the electrical characteristics of materials, which could improve the food recognition performance.

Conclusion

In conclusion, we demonstrated iEat, a novel wearable ADM paradigm with impedance sensing backed by an abstracted human-food interaction circuit model, for automatic dietary monitoring. We first explain the relationship between dietary activities and measured body impedance through an abstracted human-food interaction circuit model. Then, a wearable hardware prototype was designed to leverage this circuit model for ADM. Through a 40-meal and 10-participant experiment and machine learning, we showed that iEat could detect five dietary activities with an average micro F1-score of 86.4% and simultaneously analyze the food content through metal utensils with an average micro F1-score of 64.2%. The result shows that the use of dynamic impedance sensing provides a non-invasive and efficient method for tracking food intake activities, from preparation to ingestion, with the potential to bring significant advancements for ADM.

Methods

Topical subheadings are allowed. Authors must ensure that their Methods section includes adequate experimental and characterization data necessary for others in the field to reproduce their work.

Ethical agreement

All participants signed an informed consent following the Declaration of Helsinki. The ethical committee of Kaiserslautern University and the German Research have approved the study. Participation was entirely voluntary and could be withdrawn at any time. The participants received 15 euros as compensation for their participation. The subjects could deny answering questions if they feel uncomfortable in any way. There are no risks associated with this user study. Discomforts or inconveniences will be minor and are not likely to happen. All data provided in this user study will be treated confidentially, will be saved encrypted, and cannot be viewed by anyone outside this research project unless separate permission is signed to allow it. The data in this study will be subject to the General Data Protection Regulation (GDPR) of the European Union (EU) and treated in compliance with the GPDR.