Keywords

1 Introduction

With the spread of smartphones, users of online recipe services such as Allrecipe and Cookpad are on the rise. For example, in CookpadFootnote 1, the number of users has increased from 26 million in 2013 to almost 60 million by 2017 [1]. According to a survey by Cookpad, more than 60% of respondent have shown increment in the use of online recipes and almost half of the respondents have stopped purchasing cookbooks and magazines. This proves that there is a rapid increase in the interest of people towards using online recipes.

It is expected that online recipe service users and the number of online recipes will continue to increase. However, even if you cook in accordance with an online recipe, it does not necessarily match the preferable taste of the person who eats it because the preference of taste is different depending on the person. It is desirable to match dishes to the preferable taste as much as possible even though using online recipes because it is apparent that eating tasty dishes affects human satisfaction.

In recent years, the paradigm called IoT (Internet of Things) has become widespread. Not only smart devices such as smartphone and tablet but various items used in everyday life have now become equipped with sensors and Internet connectivity. Introduction of IoT into industrial, medical and other fields is progressing, but a movement to apply IoT to cooking utensils and the kitchen itself has also begun. For example, Pantelligent [2], a frying pan embedded with a temperature sensor and radio communication function, can provide the time to heat as well as to flip the ingredients by performing temperature management in cooperation with smartphone. As a result of the Panasonic survey [3], 76.5% of housewives have been devising to save the trouble of cooking. From these movements, it appears that various smart kitchen utensils with IoT will spread to homes in the future. However, there is no cooking utensil focusing on adjusting the amount of seasoning depending on the user’s preference.

As a preliminary experiment, we investigated about rough estimation and preference. The result showed that even though the subject user cooked on a regular basis, the difference between rough estimation and preferable amount of seasoning was about 35%. Moreover, we served instant soups with different amount of hot water to the 24 participants, by letting them choose their favorite from the soups. As a result, 12 participants chose strong soups, 11 participants chose weak soups, and only one participant chose the soup with the appropriate amount written on the official package. From these results, we assume that it is useful to realize a method that brings the taste close to the user’s preference and supports to add the precise amount of seasoning.

In this paper, we propose a cooking support system that adjusts the amount of seasoning for a recipe depending on user’s preference and supports adding seasoning by Smart Cruet. The system has three main functions: (1) to extract a user’s preference by learning the assessment of individual tasty for the consumed amount of seasoning; (2) to determine the amount of seasoning depending on user’s preference; (3) to measure and notify the input amount of seasoning by Smart Cruet in real time. Our system analyzes the user’s preference for five basic tastes, such as saltiness, sweetness, sourness, bitterness, and umami (savoriness) [4, 5], based on feedback according to the five-grade evaluation for each meal, and to adjust the amount of seasonings depending on the user’s preference. In addition, the system facilitates to add the seasonings by Smart Cruet. We assume that the users of our system can improve their satisfaction with eating, which also serves as a factor to improve their QoL. We conducted two experiments to know the user’s preference. At first, we served a dish cooked in accordance with the original recipe, and then, collected the feedback according to the five grades evaluation, such as very strong, strong, favorite, weak and very weak for the dish. Then, in the next step, we served a dish adjusted based on the feedback and collected the feedback for the dish again. As a result, we confirmed that approximately half of subjects who did not evaluate ‘favorite’ to the first dish evaluated ‘favorite’ for the adjusted dish. For the second experiment, we conducted an experiment for 14 days to confirm determine how many days are needed to bring the adjustment model close to the user’s preferable taste. We do so because the adjustment based on one-time feedback in the first experiment was not able to reach the preference level of all the participants. As a result, we found that it took approximately seven days to bring the taste close to the user’s preference.

2 Related Works

2.1 Food Recognition

Mirtchouk et al. [6] proposed a food recognition method, which uses audio sensor and accelerometers from smart watch and Google Glass to automatically detect food type and amount. This method can classify food into 40 types and estimate consumed amount by mastication sound and arm and head actions. Another device, CogKnife [7], a knife equipped with a microphone, recognizes food types from the cutting sounds. This method can classify food into six types with 96% accuracy from the cutting sounds by a classification model based on SVM (Support Vector Machine). These methods mainly recognize the food type but do not support to recognize the seasoning type and to estimate the amount of seasonings.

FoodCam [8] is a food recognition system by using camera of smartphone. This system classifies a food photo into five food types, from 100 food types with 79% accuracy by a classification model based on SVM. Bettadapura et al. [9] also proposed a food recognition system which recognizes the types of restaurant and food by location context and photo from smartphone. These systems can recognize the food type from a food photo but cannot identify the types of consumed seasonings and also cannot estimate the amount of the seasonings.

2.2 Food Recommendation

Kadowaki et al. [10] proposed a recipe recommendation system by using tweets on Twitter. The system analyzes user’s preference based on the eaten food within 3 h from posting of the tweet, and recommends a suitable recipe by estimation of current context from the latest tweet. Li et al. [11] also proposed a recipe recommendation algorithm by combining content-based filtering and collaborative filtering algorithms. The system analyzes user’s preference with the recipes evaluated as Light or Piquancy to recommend the suitable recipes. Furthermore, Yamamoto et al. [12] proposed a recipe recommendation system based on cooking history. Hence, there are many studies about recipe recommendation however these studies don’t focus on the adjustment of seasoning, thus, users can only select from the recommended recipes, that is, users cannot eat dishes that the users want to eat based on their preferable taste.

2.3 Commercial Smart Kitchen Utensils

Pantelligent [2], a frying pan with a temperature sensor, realizes the proper doneness for each food ingredient. This utensil shows the proper time to heat the food ingredient and the proper timing to flip the food ingredient, to a smartphone by the management of the temperature on the frying pan. Another device, Grill alert [13] measures the temperature of grill and can notify the proper doneness to smartphone. However, these utensils do not support to show the proper amount of seasonings. Crock pot, a cooking pot [14] embedded with a temperature sensor and radio communication function, can detect the temperature of the cooking pot and then adjust to the proper temperature but cannot adjust the taste of food.

There has been lot of studies about food recognition, recipe recommendation, and smart kitchen utensils. However, there is no study to adjust the amount of seasonings from a recipe depending on the user’s preferable taste. Therefore, we propose a cooking support system, which (1) analyzes the user’s preference for the five basic tastes [4, 5], such as saltiness, sourness, bitterness, sweetness, and umami (savoriness), (2) adjusts the amount of seasoning depending on the user’s preference, and (3) uses a LED light embedded in the Smart Cruet to indicate the moment to stop adding the seasoning.

3 Preliminary Experiments

3.1 Investigation of Individual Differences in Taste Preference

In order to find out, whether there are individual differences in user,s preference, we conducted an experiment in which the participants eat miso soups with different amounts of hot water, and then choose the most favorite soup. In this investigation, we prepared the instant miso soups with five different amounts of hot water, such as 128 ml (−20%), 144 ml (−10%), 160 ml (correct amount on the recipe), 176 ml (+10%) and 192 ml (+20%). To avoid bias from prior knowledge, we didn’t tell the participants the amount of hot water in each soup.

The participants rated each soup according to their personal taste. Table 1 shows the count of favorites for each soup. 12 participants answered that the soups with 128 ml and 144 ml, which are stronger soups than the correct soup with 160 ml, are the most favorite. On the other hand, 11 participants answered that 176 ml and 192 ml, which are weaker soups, are the most favorite. There was only one participant answered that the correct amount of soup with 160 ml is the most favorite soup. From this result, we found that there are the individual differences in user’s preference even though the food is too strong/weak like the case of differing with 20% than the correct amount defined by the manufacturer.

Table 1. Individual Differences in user’s preference for miso soups with different amount of hot water
Full size table

3.2 Investigation of Error in the Input Amount by Adding Seasoning Based on Rough Estimation

Over half of Japanese housewives add seasoning based on rough estimation. Adding seasonings based on rough estimation opens the possibility that the taste of the dish differs greatly from the intended taste of the cook or of the eater’s preference due to a too high or small amount of seasoning in comparison with the recipe. Therefore, we conducted an experiment to confirm the error between the input amount based on a rough estimation and the specific amounts. In this experiment, we prepared seven cups written the input amount of seasoning soy sauce such as large spoon: ‘1 (, 2, 3, or 4)’, and ‘small spoon: 1 (, 2, or 3)’. One measure of a large spoon represents 15 ml, and the measure of one small spoon represents 5 ml. We let 15 participants add soy sauce seasoning based on rough estimation to each cup. Among the 15 participants, six participants have been cooking more than three days in a week, the other nine participants usually cook less than three days in a week.

Fig. 1.
figure 1

Difference between rough estimation and preferable amount of seasoning in six participants with daily cooking habits (Points: Amount based on Rough Estimation, Led Line: Preferable Amount)

Full size image
Fig. 2.
figure 2

Difference between rough estimation and preferable amount of seasoning in six participants with cooking habit (Points: Amount based on Rough Estimation, Led Line: Preferable Amount)

Full size image

Figure 1a and b show the difference between rough estimation and preferable amount as stated in the recipe for the small and large spoon measure, for the nine participants, which did not cook very frequently. In the figures, the points represent the amount of seasoning with rough estimation and the red line represents the preferable amount of seasoning with measuring spoons. For these participants, the average error for small spoon was 71.8% and the average error for large spoon was 41.4%. Both of the average errors were understandably large because these participants don’t cook very frequently. On the other hand, Fig. 2a and b show the difference for the six participants, who cook frequently. For these participants, the average error for small spoon was 46.2% and the average error for large spoon was 31.8%. Even if the dish is cooked using the recipe, the dish does not always match the user’s preference because of the measuring error of the seasoning which differs from the recipe.

From these results, we assume that a system is needed to be able to adjust the amount of seasoning depending on the user’s preference in order to serve the dish to satisfy user’s preference, and assist putting seasonings correctly.

4 Cooking Support System for Seasoning with Smart Cruet

4.1 System Architecture

The system architecture for the proposed system is shown in Fig. 3. The system consists of a smartphone application, a server and Smart Cruet. Users can check details about the dish and cooking process by using their smartphones. Users are required to evaluate the dish after eating, which is then stored in the user preference database. The server also consists of recipe database, a preference analysis mechanism, and amount adjustment mechanism. Smart Cruet is used to add salt and soy sauce adjusted according to user’s preferable taste.

Fig. 3.
figure 3

System architecture

Full size image

4.2 Smartphone Application

Figure 4a shows the details of the cooking process and recipes for the dish. The smartphone application is used to record feedbacks of users which are recorded as evaluation data to the server. The smartphone application also displays recipe data and user’s preference. The users submit their evaluation based on the five basic tastes (Sweet, Salty, Sour, Bitter, Umami (savoriness)) after each meal as shown in Fig. 4b. Figure 4c shows a preference display function for the five basic tastes created for each user using their feedbacks.

Fig. 4.
figure 4

Main functions of smartphone application

Full size image

4.3 Preference Analysis Mechanism

The preference analysis mechanism creates a preference model after analyzing the evaluation data received from the users. This model is further used in amount adjustment mechanism, to determine the appropriate amount of seasonings for each user. To create a preference model, we analyze the evaluation feedback received after each meal from the user and calculate the preference model value using the following formula:

$$\begin{aligned} M_{i}=\sum ^{i-1}_{k=1}(M_{k}-F_{k})\,/\,i \\ (M_1 = 0,\ F_1 = 0,\ -2 \le F_k \le 2, \ i > 1) \end{aligned}$$

\(M_i\) and \(F_i\) represent the preference model value and feedback value for each day i, respectively.

4.4 Amount Adjustment Mechanism

The amount adjustment mechanism is used for adjusting the amount of seasoning according to the user’s preference. By using the preference model developed, the appropriate amount of seasoning for each user is derived using the following formula:

$$\begin{aligned} Adjusted\,amt. = M_i\times 0.1 \times Amt.\,of\,seasoning \end{aligned}$$

This adjusted amount is added to the current amount of seasoning used in the recipe.

4.5 Smart Cruet

Smart Cruet is developed to make it easier for users to add amount of seasonings according to their preferable taste. The Smart Cruet, as shown in Fig. 5, is composed of an acceleration sensor, a gyro sensor, a Bluetooth low energy (BLE) module, a battery, and a light emitting diode (LED). While using Smart Cruet, the embedded LED blinks to indicate that the appropriate amount of seasoning has been added and hence the user can now stop adding the seasoning to the dish.

Fig. 5.
figure 5

Smart cruet prototype

Full size image

4.6 Recipe Dataset

Since our aim is to analyze user’s preferable taste and adjust the amount of seasoning according to that taste, we required recipes that have not been affected by user’s preference. Therefore, we used Bob & Angie [15] dataset provided by OGIS-RI Co., LtdFootnote 2. We believe that the dataset is suitable as a basic recipe dataset because it contains four thousand recipes supervised with emphasis on health and nutrition by registered dietitians.

5 Experiment

Experiments were conducted to determine if the proposed system could be used to get the amount of seasoning closer to the user’s preferable taste. We also conducted experiments to check the accuracy of the Smart Cruet in real time.

5.1 Confirming Effect of One Feedback

Experimental Method. On the first day, we gathered feedbacks from 15 participants for Japanese Beef and Potato Stew dish, as shown in Fig. 6a. On the second day, the participants were asked to rate the dish prepared based on their feedbacks earlier on the level of saltiness from scale 1–5. Participants then tried the dish Boiling Spinach with Tube-shaped Fish Paste as shown in Fig. 6b in which soy sauce and salt were adjusted based on feedback received on the first day. This helped us to determine if we can approximate the preference level of users based on the feedbacks received after each meal.

Fig. 6.
figure 6

Dishes before and after adjustment

Full size image

Experimental Result. Figure 7 shows the number of feedbacks received from the 10 participants before the adjustment of saltiness, while Fig. 8 depicts the number of feedbacks received after adjustment. The feedback for the amount of seasonings is depicted from scale 1 to 5, with 1 indicating the lowest amount of saltiness, 5 indicating the highest amount, and 3 indicating the optimal amount. The feedback before the adjustment showed that out of all the participants, optimal rating i.e. 3 was provided by 5, two participants rated the seasoning as 1, while the highest saltiness scale was 4 as provided by three participants. After the adjustment, four participants rated the seasoning as optimal i.e. 3. While five participants rated the seasoning in the scale of 2, only one participant provided the rating scale of 5. Table 2 shows that we were able to increase the preferable taste to the optimal amount for four participants. We were also able to improve the taste for one participant, up to the scale of 2.

Table 2. Evaluation before and after adjustment (bold columns represent the participants who could be closer to themselves preference)
Full size table
Fig. 7.
figure 7

Feedback from 15 participants before adjustment

Full size image

Consideration. Four out of 10 participants rated the taste of the seasoning to be optimal, while six participants did not. We believe that the results from one-day experiment explained above were inconclusive. Therefore, we decided to test the efficiency of our system for a longer period of time.

5.2 Confirming Effects of Feedback After Two Weeks

Experimental Method. We conducted this experiment to investigate personal preferences over a long term period. The first day, the user cooks following a basic recipe. The user evaluates five levels of the introduced five basic tastes after the meal. Based on the feedback, the proposed system adjusts the amount of seasoning on on the second day. The users evaluate again their meal but this time with the adjusted amount of seasonings based on user’s preference. This process was repeated for 14 days, each time taking the new feedback into account. We evaluate the proposed system based on how close it approaches the user’s preference.

Experimental Result. Figure 9 shows the preference model for the five different tastes evaluated, which greatly changed until the seventh day except for bitter taste. However, the preference model on the final day did not vary much from the preference model that was created at the 7th day. It can be seen that a person’s taste preference could be modeled and improved to some extent in seven days from the start of the experiment.

Fig. 8.
figure 8

Feedback from 10 participants after adjustment

Full size image
Fig. 9.
figure 9

Transition of feedback for five basic tastes during two weeks

Full size image

Consideration. Since the preference model estimation in two days was not able to approach the majority or more of the participants, we decided that the period was not sufficiently long. Therefore, the period was extended. After conducting the additional experiments, the preference model estimation in a two week time period was able to estimate roughly the individual preference model. However, since there was only one participant in the preference model estimation experiment in the two week period, there was a possibility that it is not representative. In the future, we think that it is necessary to verify the preference analysis time period by increasing the number of participants and estimating different preference models.

5.3 Confirming Accuracy of Smart Cruet

To confirm the accuracy of adding seasoning with the Smart Cruet, we conducted an experiment. In this experiment, we added 50 ml of soy sauce 10 times using the Smart Cruet. After this we compared the amount estimated by the Smart Cruet to the actual added amount. As a result, the average of the added amount was 51.37 ml, the average of the estimated amount was 54.23 ml, thus, the average error between the actual amount and the estimated amount was 5.56%. Moreover, the error between the average estimated amount and the preferable amount (50 ml) was 8.46%.

As we showed in Sect. 3.2, even if the cooks are experienced, there is approximately 30% error between rough estimation and preferable amount. Therefore, it is assumed that a dish seasoned by the Smart Cruet is closer to user’s preference than by using rough estimation. Furthermore, it is easier to use than using measuring spoons.

6 Conclusion

In this paper, we proposed a cooking support system that adjusts the amount of seasoning depending on the user’s preference and support adding the correct amount of seasoning, through smartphone and Smart Cruet. The system analyzes the user’s preference based on the feedback for five basic tastes evaluated by five grades after each eating a dish and adjust the amount of seasoning in the selected recipe depending on the user’s preference. For the adjusted amount of seasoning, Smart Cruet, which embeds motion sensors, LED light, and BLE communication interface, indicates the moment to stop adding the seasoning. We were successful in improving the adjustment to user’s preference in 7 days through a two week feedback experiment. Furthermore, Smart Cruet achieved a 5.56% average error, which is clearly lower than adding seasoning based on rough estimation. Because we used the recipes made by registered dietitians in this paper, as future work, we will mainly focus on adjusting the amount of seasoning for online recipes registered by non professional cooks. We also plan to test the efficiency of our system with higher number of participants.