WO2022074754A1

WO2022074754A1 - Information processing method, information processing system, and program

Info

Publication number: WO2022074754A1
Application number: PCT/JP2020/037966
Authority: WO
Inventors: 竜之介大道; 慶二郎才野; 正宏清水
Original assignee: ヤマハ株式会社
Priority date: 2020-10-07
Filing date: 2020-10-07
Publication date: 2022-04-14
Also published as: JPWO2022074754A1; CN116324965A

Abstract

This invention includes: editing, in accordance with a first instruction from a user, first time-series data representing a time series of a feature amount of a sound produced by pronouncing a symbol string with a first pronunciation style; saving, as a new version of data, first historical data corresponding to the edited first time-series data, for each edit of the first time-series data; editing, in accordance with a second instruction from the user, second time-series data representing a time series of a feature amount of a sound produced by pronouncing the symbol string with a second pronunciation style which is different from the first pronunciation style; saving, as a new version of data, second historical data corresponding to the edited second time-series data, for each edit of the second time-series data; and acquiring first time-series data corresponding to a first historical data piece, among a plurality of saved first historical data pieces of different versions, that corresponds to an instruction from the user, or second time-series data corresponding to a second historical data piece, among a plurality of saved second historical data pieces of different versions, that corresponds to an instruction from the user.

Description

Information processing methods, information processing systems and programs

This disclosure relates to the processing of time series data.

Various speech synthesis techniques for synthesizing speeches of arbitrary phonology have been proposed conventionally. For example, Patent Document 1 discloses a technique for synthesizing a singing voice that pronounces a note sequence instructed by a user on an editing screen. The edit screen is a piano roll screen in which the time axis and the pitch axis are set. The user specifies a phonetic (phonetic character), a pitch, and a pronunciation period for each note that constitutes a musical piece.

Japanese Unexamined Patent Publication No. 2016-90916

In order to synthesize a voice that accurately reflects the user's intention, the user is required to perform trial and error by repeating editing of voice synthesis conditions (for example, various parameters) and listening to the actual voice. A configuration that allows the process of canceling the latest edit in reverse order (undo) or the process of re-executing the canceled edit (redo) among multiple edits instructed by the user in order is also assumed, but a simple undo Or, it is actually difficult for the user to instruct editing by trial and error while comparing the results of various editing with each other only by redoing. In the above description, the scene of speech synthesis is illustrated, but similar problems are assumed in various scenes of generating time-series data. In consideration of the above circumstances, it is an object of the present disclosure to facilitate the generation of time-series data in line with the user's intention.

In order to solve the above problems, the information processing method according to one aspect of the present disclosure uses the first time-series data representing the time-series of the feature amount of the sound in which the symbol string is pronounced in the first pronunciation style. Edit according to the first instruction from, and for each edit of the first time series data, the first history data corresponding to the first time series data after the edit is saved as a new version of the data, and the first The second time-series data representing the time series of the feature amount of the sound that pronounced the symbol string in the second pronunciation style different from the first pronunciation style is edited according to the second instruction from the user, and the second Each time the time series data is edited, the second history data corresponding to the edited second time series data is saved as new version data, and among the plurality of first history data of the saved different versions. The first time-series data corresponding to the first history data in response to the instruction from the user, or the second of the plurality of saved second history data of different versions in response to the instruction from the user. 2 Acquire the second time-series data corresponding to the historical data.

The information processing system according to one aspect of the present disclosure edits the first time-series data representing the time-series of the feature amount of the sound that pronounces the symbol string in the first pronunciation style according to the first instruction from the user. Then, the second time series data representing the time series of the feature amount of the sound that pronounced the symbol string in the second pronunciation style different from the first pronunciation style is edited according to the second instruction from the user. Each time the editing processing unit edits the first time-series data, the first history data corresponding to the edited first time-series data is saved as new version data, and the second time-series data is edited. Each time, it is provided with an information management unit that saves the second history data corresponding to the edited second time-series data as new version data, and the information management unit has the saved different versions. The use of the first time-series data corresponding to the first history data according to the instruction from the user among the plurality of first history data, or the second history data of a plurality of different versions of the saved data. Acquire the second time-series data corresponding to the second history data according to the instruction from the person.

The information processing system according to one aspect of the present disclosure edits the first time-series data representing the time-series of the feature amount of the sound that pronounces the symbol string in the first pronunciation style according to the first instruction from the user. Then, the second time series data representing the time series of the feature amount of the sound that pronounced the symbol string in the second pronunciation style different from the first pronunciation style is edited according to the second instruction from the user. For each editing of the editing processing unit and the first time-series data, the first history data corresponding to the edited first time-series data is saved as new version data, and the second time-series data is stored. A program that causes the computer system to function as an information management unit that saves the second history data corresponding to the edited second time-series data as new version data for each edit, and the information management unit is a program. Among the plurality of saved different versions of the first history data, the first time-series data corresponding to the first history data in response to the instruction from the user, or the plurality of saved different versions of the first history data. Of the second history data, the second time-series data corresponding to the second history data according to the instruction from the user is acquired.

It is a block diagram which illustrates the structure of the information processing system which concerns on 1st Embodiment. It is a schematic diagram of an edit screen. It is a block diagram which illustrates the functional structure of an information processing system. It is a flowchart illustrating the procedure of the 1st editing process. It is a flowchart which illustrates the procedure of the 2nd editing process. It is a flowchart which illustrates the procedure of the 3rd editing process. It is explanatory drawing of the data structure in a history area. It is a flowchart illustrating the procedure of the 1st management process. It is a flowchart which illustrates the procedure of the 2nd management process. It is a flowchart which illustrates the procedure of the 3rd management process. It is a schematic diagram of the edit screen in 2nd Embodiment. It is a block diagram which illustrates the functional structure of the information processing system in 2nd Embodiment. It is explanatory drawing of the data structure in the history area in 2nd Embodiment. It is a schematic diagram of a comparison screen. It is explanatory drawing of the synthetic sound in 3rd Embodiment. It is a schematic diagram of the edit screen in 3rd Embodiment. It is a schematic diagram of an edit screen in a modification.

A: First Embodiment FIG. 1 is a block diagram illustrating the configuration of the information processing system 100 according to the first embodiment of the present disclosure. The information processing system 100 is an acoustic processing system that generates an acoustic signal Z. The acoustic signal Z is a signal in the time domain representing the waveform of the synthetic sound. The synthetic sound is, for example, a musical instrument sound produced by a virtual performer playing a musical instrument, or a singing sound produced by, for example, a virtual singer singing a song.

The information processing system 100 is realized by a computer system including a control device 11, a storage device 12, a sound emitting device 13, a display device 14, and an operating device 15. The information processing system 100 is realized by, for example, an information device such as a smartphone, a tablet terminal, or a personal computer. The information processing system 100 is realized not only by a single device but also by a plurality of devices (for example, a client-server system) configured as separate bodies from each other.

The control device 11 is a single or a plurality of processors that control each element of the information processing system 100. Specifically, for example, one or more types of processors such as CPU (Central Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit). 3. The control device 11 is configured. The control device 11 executes various processes for generating the acoustic signal Z.

The storage device 12 is a single or a plurality of memories for storing a program executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a known recording medium such as a magnetic recording medium or a semiconductor recording medium. The storage device 12 may be composed of a combination of a plurality of types of recording media. Further, a portable recording medium attached to and detached from the information processing system 100, or a recording medium capable of writing and reading via a communication network (for example, cloud storage) may be used as the storage device 12.

The sound emitting device 13 reproduces the synthetic sound represented by the acoustic signal Z generated by the control device 11. The sound emitting device 13 is, for example, a speaker or headphones. The D / A converter that converts the acoustic signal Z from digital to analog and the amplifier that amplifies the acoustic signal Z are not shown for convenience. Further, in FIG. 1, the configuration in which the sound emitting device 13 is mounted on the information processing system 100 is illustrated, but the sound emitting device 13 separate from the information processing system 100 is connected to the information processing system 100 by wire or wirelessly. May be done.

The display device 14 displays an image under the control of the control device 11. The display device 14 is composed of a display panel such as a liquid crystal panel or an organic EL (ElectroLuminescence) panel. The operation device 15 is an input device that receives instructions from the user. The operation device 15 is, for example, a plurality of controls operated by the user or a touch panel for detecting contact by the user. The user can instruct the condition of the synthesized sound by operating the operation device 15. The display device 14 displays an image (hereinafter referred to as “editing screen”) G referred to by the user for instructing the condition of the synthetic sound.

FIG. 2 is a schematic diagram of the edit screen G. The editing screen G includes a plurality of editing areas E (En, Ef, and Ew). A common time axis (horizontal axis) is set in the plurality of editing areas E. The section of the synthetic sound displayed on the edit screen G is changed according to the instruction from the user to the operation device 15.

In the editing area En, a time series (hereinafter referred to as "note sequence") N of a plurality of notes constituting the score of the synthesized sound is displayed. A coordinate plane defined by a time axis and a pitch axis (vertical axis) is set in the editing area En. An image representing each note constituting the note sequence N is arranged in the editing area En. A pitch (for example, a note number) and a pronunciation period are specified for each note in the note sequence N. When the synthetic sound is a singing sound, the phoneme is specified for each note. In the editing area En, performance symbols such as crescendo, forte, and decrescendo are also displayed. The user can give an edit instruction Qn to the edit area En by operating the operation device 15. The edit instruction Qn is an instruction to edit the note string N. Specifically, the edit instruction Qn is an instruction to add or delete each note in the note sequence N, an instruction to change the condition (pitch, pronunciation period or phonology) of each note, or an instruction to change the performance symbol. be.

In the editing area Ef, a time series (hereinafter referred to as "feature column") F of the feature amount of the synthetic sound is displayed. The feature amount is an acoustic feature amount of the synthetic sound. Specifically, the feature column F (that is, the temporal transition of the fundamental frequency) is displayed in the editing area Ef with the fundamental frequency (pitch) of the synthesized sound as the feature amount. The user can give an edit instruction Qf to the edit area Ef by operating the operation device 15. The edit instruction Qf is an instruction to edit the feature column F. Specifically, the editing instruction Qf is, for example, an instruction for changing the time change of the feature amount in the desired section of the feature column F displayed in the editing area Ef.

In the editing area Ew, the waveform W of the synthesized sound on the time axis is displayed. The user can give an edit instruction Qw to the edit area Ew by operating the operation device 15. The edit instruction Qw is an instruction to edit the waveform W. Specifically, the editing instruction Qw is an instruction to change the waveform in the user's desired section of the waveform W displayed in the editing area Ew.

The editing screen G includes, in addition to the plurality of editing areas E exemplified above, a plurality of operating areas (Gn, Gf and Gw) corresponding to different editing areas E, and an operating image B1 (playback). The operation image B1 is a software button that can be operated by the user using the operation device 15. Specifically, the operation image B1 is an operation element for the user to instruct the reproduction of the synthesized sound. Specifically, when the user operates the operation image B1, the synthetic sound of the waveform W displayed in the editing area Ew is reproduced from the sound emitting device 13.

The operation area Gn is an area related to the note string N. Specifically, the note string version number Vn, the operation image Gn1 and the operation image Gn2 are displayed in the operation area Gn. The note string version number Vn is a number representing the version of the note string N displayed in the editing area En. The note string version number Vn is incremented by 1 each time the note string N is edited according to the edit instruction Qn. Further, the user can change the note string version number Vn in the operation area Gn to an arbitrary numerical value by operating the operation device 15. Of the plurality of versions of the note string N generated in the process of past editing, the note string N of the version corresponding to the note string version number Vn changed by the user is displayed in the editing area En.

The operation image Gn1 and the operation image Gn2 are software buttons that can be operated by the user using the operation device 15. The operation image Gn1 is an operation element for the user to instruct to return the note string N to the state before the execution of the immediately preceding edit (Undo). That is, when the user operates the operation image Gn1, the note string version number Vn is changed to the immediately preceding numerical value, and the note string N of the version corresponding to the changed note string version number Vn is the edit area En. Is displayed in. Therefore, the operation image Gn1 is also expressed as an operator for retreating the note string version number Vn to the immediately preceding numerical value (that is, canceling the immediately preceding edit regarding the note string N). On the other hand, the operation image Gn2 is an operator for instructing the user to perform the editing canceled by the operation on the operation image Gn1 again (Redo).

The operation area Gf is an area related to the feature column F. Specifically, the feature column version number Vf, the operation image Gf1, and the operation image Gf2 are displayed in the operation area Gf. The feature column version number Vf is a number representing the version of the feature column F displayed in the editing area Ef. The feature column version number Vf is incremented by 1 each time the feature column F is edited according to the edit instruction Qf. Further, the user can change the feature column version number Vf in the operation area Gf to an arbitrary numerical value by operating the operation device 15. Of the plurality of versions of the feature column F generated in the process of past editing, the feature column F of the version corresponding to the feature column version number Vf changed by the user is displayed in the editing area Ef.

The operation image Gf1 and the operation image Gf2 are software buttons that can be operated by the user using the operation device 15. The operation image Gf1 is an operation element for the user to instruct to return the feature column F to the state before the execution of the immediately preceding edit (Undo). That is, when the user operates the operation image Gf1, the feature column version number Vf is changed to the immediately preceding numerical value, and the feature column F of the version corresponding to the changed feature column version number Vf is the edit area Ef. Is displayed in. Therefore, the operation image Gf1 is also expressed as an operator for retreating the feature column version number Vf to the immediately preceding numerical value (that is, canceling the immediately preceding edit regarding the feature sequence F). On the other hand, the operation image Gf2 is an operator for instructing the user to perform the editing canceled by the operation on the operation image Gf1 again (Redo).

The operation area Gw is an area related to the waveform W. Specifically, the waveform version number Vw, the operation image Gw1 and the operation image Gw2 are displayed in the operation area Gw. The waveform version number Vw is a number representing the version of the waveform W displayed in the editing area Ew. The waveform version number Vw is incremented by 1 each time the waveform W is edited according to the edit instruction Qw. Further, the user can change the waveform version number Vw in the operation area Gw to an arbitrary numerical value by operating the operation device 15. Of the plurality of versions of the waveform W generated in the process of past editing, the version of the waveform W corresponding to the waveform version number Vw changed by the user is displayed in the editing area Ew.

The operation image Gw1 and the operation image Gw2 are software buttons that can be operated by the user using the operation device 15. The operation image Gw1 is an operator for instructing the user to return the waveform W to the state before the execution of the immediately preceding edit (Undo). That is, when the user operates the operation image Gw1, the waveform version number Vw is changed to the immediately preceding value, and the waveform W of the version corresponding to the changed waveform version number Vw is displayed in the editing area Ew. To. Therefore, the operation image Gw1 is also expressed as an operator for retreating the waveform version number Vw to the immediately preceding value (that is, canceling the immediately preceding edit regarding the waveform W). On the other hand, the operation image Gw2 is an operator for instructing the user to perform the editing canceled by the operation on the operation image Gw1 again (Redo).

As described above, in the first embodiment, a plurality of version numbers V (Vn, Vf, Vw) are used. An increase in each version number (increment) means the progress of the editing work, and a decrease in each version number (decrement) means a recession in the editing work.

FIG. 3 is a block diagram illustrating a functional configuration of the information processing system 100. The control device 11 executes a program stored in the storage device 12 to perform a plurality of functions (display control unit 20, editing processing unit 30, and information) for editing synthetic sound conditions and generating an acoustic signal Z. Realize the management unit 40). The display control unit 20 causes the display device 14 to display an image under the control of the control device 11. For example, the display control unit 20 causes the display device 14 to display the editing screen G illustrated in FIG. Further, the display control unit 20 updates the edit screen G in response to an instruction (Qn, Qf or Qw) from the user.

The editing processing unit 30 in FIG. 3 edits the synthetic sound conditions (note sequence N, feature sequence F, and waveform W) according to an instruction (Qn, Qf, or Qw) from the user. The editing processing unit 30 includes a first editing unit 31, a first generation unit 32, a second editing unit 33, a second generation unit 34, and a third editing unit 35.

The first editorial unit 31 edits the note string data Dn. The note string data Dn is time-series data representing the note sequence N of the synthesized sound. Specifically, the first editing unit 31 edits the note string data Dn according to the editing instruction Qn from the user for the editing area En. The display control unit 20 displays the musical note string N represented by the musical note string data Dn edited by the first editing unit 31 in the editing area En.

The first generation unit 32 generates the feature sequence data Df from the note sequence data Dn edited by the first editing unit 31. The feature sequence data Df is time-series data representing the feature sequence F of the synthesized sound. In addition, in addition to the note data at the time point, at least the notes before and after the note are generated for the generation of the feature amount at each time point on the time axis among the plurality of feature amounts constituting the feature sequence F. The data of one note is used. That is, the feature sequence data Df is generated according to the context of the note sequence N represented by the note sequence data Dn.

Specifically, the first generation unit 32 generates the feature column data Df using the first generation model M1. The first generative model M1 is a statistical inference model that inputs the note sequence data Dn and outputs the feature sequence data Df. Specifically, the first generative model M1 is a trained model that has learned the relationship between the note sequence N and the feature sequence F. The first generative model M1 is composed of, for example, a deep neural network (DNN). For example, an arbitrary form of deep neural network such as a convolutional neural network (CNN) or a recurrent neural network (RNN) is used as the first generative model M1. In addition, additional elements such as long short-term memory (LSTM: Long Short-Term Memory) or Self-Attention may be mounted on the first generation model M1.

The first generation model M1 includes a program that causes the control device 11 to execute an operation for generating feature sequence data Df from the note sequence data Dn, and a plurality of variables (specifically, weighted values and biases) applied to the operation. It is realized by the combination of. The plurality of variables defining the first generation model M1 are preset and stored in the storage device 12 by machine learning using the plurality of first training data. Each of the plurality of first training data includes the note sequence data Dn and the feature sequence data Df (correct answer value). In the machine learning of the first generation model M1, the feature sequence data Df output by the provisional first generation model M1 for the note sequence data Dn of each first training data and the feature sequence data of the first training data. A plurality of variables of the first generation model M1 are updated iteratively so that the error with Df is reduced. Therefore, the first generative model M1 is a statistically valid feature for the unknown note sequence data Dn under the latent tendency between the note sequence N and the feature sequence F in the plurality of first training data. Output the column data Df.

The second editing unit 33 edits the feature column data Df generated by the first generation unit 32. Specifically, the second editing unit 33 edits the feature column data Df according to the editing instruction Qf from the user for the editing area Ef. The display control unit 20 displays the feature column F represented by the feature column data Df generated by the first generation unit 32 or the feature column F represented by the feature column data Df edited by the second editing unit 33 in the editing area Ef. do.

The second generation unit 34 generates waveform data Dw from the note sequence data Dn and the feature sequence data Df. The waveform data Dw is time-series data representing the waveform W of the synthesized sound. That is, the waveform data Dw is composed of a time series of a plurality of samples representing the acoustic signal Z. The acoustic signal Z is generated by D / A conversion and amplification for the waveform data Dw. The feature sequence data Df immediately after being generated by the first generation unit 32 (that is, the feature sequence data DF not edited by the second editing unit 33) may be used for generating the waveform data Dw.

The second generation unit 34 generates waveform data Dw using the second generation model M2. The second generative model M2 is a statistical inference model that outputs waveform data Dw by inputting a set of note sequence data Dn and feature sequence data Df (hereinafter referred to as “input data Din”). Specifically, the second generative model M2 is a trained model in which the relationship between the set of the note sequence N and the feature sequence F and the waveform W is learned. The second generative model M2 is composed of, for example, a deep neural network. For example, an arbitrary form of deep neural network such as a convolutional neural network or a recurrent neural network is used as the second generative model M2. In addition, additional elements such as long-term memory or self-attention may be mounted on the second generative model M2.

The second generation model M2 is a program that causes the control device 11 to execute an operation of generating waveform data Dw from the input data Din including the note string data Dn and the feature sequence data Df, and a plurality of variables applied to the operation (the second generation model M2). Specifically, it is realized in combination with a weighted value and a bias). The plurality of variables defining the second generation model M2 are preset and stored in the storage device 12 by machine learning using the plurality of second training data. Each of the plurality of second training data includes input data Din and waveform data Dw (correct answer value). In the machine learning of the second generative model M2, the waveform data Dw output by the provisional second generative model M2 with respect to the input data Din of each second training data and the waveform data Dw of the second training data. A plurality of variables of the second generative model M2 are updated iteratively so that the error is reduced. Therefore, the second generative model M2 is statistical with respect to the unknown input data Din under the latent tendency between the set of the note sequence N and the feature sequence F and the waveform W in the plurality of second training data. Outputs appropriate waveform data Dw.

The third editing unit 35 edits the waveform data Dw generated by the second generation unit 34. Specifically, the third editing unit 35 edits the waveform data Dw according to the editing instruction Qw from the user for the editing area Ew. The display control unit 20 displays the waveform W represented by the waveform data Dw generated by the second generation unit 34 or the waveform W represented by the waveform data Dw edited by the third editing unit 35 in the editing area Ew. Further, when the operation image B1 (reproduction) is operated by the user, the acoustic signal Z corresponding to the waveform data Dw generated by the second generation unit 34 or the waveform data Dw edited by the third editing unit 35 is emitted. By being supplied to the device 13, the synthesized sound is reproduced.

The information management unit 40 manages versions of each of the note sequence data Dn, the feature sequence data Df, and the waveform data Dw. Specifically, the information management unit 40 manages the note sequence version number Vn, the feature sequence version number Vf, and the waveform version number Vw.

Further, the information management unit 40 stores different versions of data (hereinafter referred to as “history data”) for each of the note sequence data Dn, the feature sequence data Df, and the waveform data Dw in the storage device 12. A history area and a work area are set in the storage device 12. The history area is a storage area in which the history of editing related to the synthetic sound condition is stored. On the other hand, the work area is a storage area in which the note sequence data Dn, the feature sequence data Df, and the waveform data Dw are temporarily stored in the process of editing using the edit screen G.

Specifically, the information management unit 40 saves the edited note sequence data Dn as the first history data Hn [Vn, Vf, Vw] in the history area for each edit of the note sequence N in response to the edit instruction Qn. do. That is, the new version of the note string data Dn is stored in the storage device 12 as the first history data Hn [Vn, Vf, Vw].

Further, the information management unit 40 saves the second history data Hf [Vn, Vf, Vw] corresponding to the edited feature column data Df according to the edit instruction Qf in the history area as new version data. The second history data Hf [Vn, Vf, Vw] of the first embodiment is data showing how the feature column data Df was edited according to the edit instruction Qf (that is, the time series of the edit instruction Qf). The second history data Hf [Vn, Vf, Vw] is also referred to as data representing the difference between the feature column data Df before and after editing.

Similarly, the information management unit 40 saves the third history data Hw [Vn, Vf, Vw] corresponding to the edited waveform data Dw according to the edit instruction Qw in the history area as new version data. The third history data Hw [Vn, Vf, Vw] of the first embodiment is data showing how the waveform data Dw was edited according to the editing instruction Qw (that is, the time series of the editing instruction Qw). The third history data Hw [Vn, Vf, Vw] is also referred to as data representing the difference between the waveform data Dw before and after editing.

4 to 6 are flowcharts illustrating a specific procedure of the editing process Sa (Sa1, Sa2 and Sa3) for editing the condition of the synthetic sound according to the editing instruction Q (Qn, Qf or Qw) from the user. Is. FIG. 4 is a flowchart of the first editing process Sa1 relating to the editing of the note string N. The first editing process Sa1 is started with the editing instruction Qn for the note string N as a trigger. When the first editing process Sa1 is started, the first editing unit 31 edits the current note string data Dn according to the editing instruction Qn (Sa101).

The information management unit 40 increases the note string version number Vn by "1" (Sa102). When the edit instruction Qn is first given, the note string data Dn is newly generated (Sa101), and the note string version number Vn is initialized to "0" (Sa102). Further, the information management unit 40 initializes the feature column version number Vf to "0" (Sa103) and initializes the waveform version number Vw to "0" (Sa104). Then, the information management unit 40 uses the note string data Dn edited by the first editing unit 31 as the first history data Hn [Vn, Vf = 0, Vw = 0] of the note string N in the history area of the storage device 12. Save to (Sa105).

As can be understood from the above explanation, for each edit of the note string data Dn according to the edit instruction Qn, the note string data Dn of the edited version is the first history data Hn [Vn, Vf = 0, Vw = 0. ] Is saved in the history area (Sa105), the note string version number Vn is increased (Sa102), and the feature column version number Vf and the waveform version number Vw are initialized (Sa103 and Sa104).

The first generation unit 32 generates the feature sequence data Df by supplying the note sequence data Dn edited by the first editing unit 31 to the first generation model M1 (Sa106). The feature sequence data Df generated by the first generation unit 32 is stored in the work area of the storage device 12. Further, the second generation unit 34 supplies the input data Din including the note sequence data Dn edited by the first editing unit 31 and the feature sequence data Df generated by the first generation unit 32 to the second generation model M2. This generates waveform data Dw (Sa107). The waveform data Dw generated by the second generation unit 34 is stored in the work area of the storage device 12.

Note that the note string data Dn requires one data for each note. The feature sequence data Df is composed of one sample every several milliseconds to several tens of milliseconds in order to represent the change in pitch in each note. Since the waveform data Dw represents the waveform of each note, one sample is configured for each sampling period (for example, 1/50 kHz to 20 μsec). As shown in the above example, the amount of data of the feature sequence data Df created from one note sequence data Dn is several hundred to several thousand times the amount of data of the note sequence data Dn, and one feature sequence. The amount of data of the waveform data Dw generated from the data Df is several hundred times to several thousand times the amount of data of the feature column data Df. In consideration of the above circumstances, in the first embodiment, the upper layer data (note string data Dn) is stored as it is as the first history data Hn [Vn, Vf = 0, Vw = 0]. On the other hand, since the layer data (feature column data Df and waveform data Dw) has a large amount of data as described above, only the difference from the upper layer data is stored as historical data. According to the above configuration, there is an advantage that the amount of data stored in the storage device 12 can be significantly reduced with respect to the hierarchical data as compared with the configuration in which the data itself is stored.

The display control unit 20 updates the edit screen G (Sa108-Sa110). Specifically, the display control unit 20 displays the note string N represented by the note string data Dn edited by the first editing unit 31 in the editing area En (Sa108). Further, the display control unit 20 displays the feature column F represented by the current feature column data Df stored in the work area in the edit area Ef (Sa109). Similarly, the display control unit 20 displays the waveform W represented by the current waveform data Dw stored in the work area in the edit area Ew (Sa110).

FIG. 5 is a flowchart of the second editing process Sa2 relating to the editing of the feature column F. The second editing process Sa2 is started with the editing instruction Qf for the feature column F as a trigger. When the second editing process Sa2 is started, the second editing unit 33 edits the current feature column data Df according to the editing instruction Qf (Sa201).

The information management unit 40 increases the feature column version number Vf by "1" (Sa202). Further, the information management unit 40 maintains the note string version number Vn at the current value Cn (Sa203) and initializes the waveform version number Vw to “0” (Sa204). Then, the information management unit 40 saves the second history data Hf [Vn, Vf, Vw = 0] representing the editing instruction Qf this time in the history area as new version data (Sa205).

As can be understood from the above explanation, every time the feature column data Df is edited according to the edit instruction Qf, the second history data Hf [Vn, Vf, Vw = 0] corresponding to the edited feature column data Df is generated. It is saved in the history area (Sa205), the note sequence version number Vn is maintained (Sa203), the feature sequence version number Vf is increased (Sa202), and the waveform version number Vw is initialized (Sa204). Note that step Sa203 may be omitted.

The second generation unit 34 generates waveform data Dw by supplying input data Din including the current note sequence data Dn and the feature sequence data Df edited by the second editing unit 33 to the second generation model M2. (Sa206). The waveform data Dw generated by the second generation unit 34 is stored in the work area of the storage device 12.

The display control unit 20 updates the edit screen G (Sa207 and Sa208). Specifically, the display control unit 20 displays the feature column F represented by the feature column data Df edited by the second editing unit 33 in the editing area Ef (Sa207). Further, the display control unit 20 displays the waveform W represented by the current waveform data Dw stored in the work area in the edit area Ew (Sa208). In the second editing process Sa2, the note string N in the editing area En is not updated.

FIG. 6 is a flowchart of the third editing process Sa3 relating to the editing of the waveform W. The third editing process Sa3 is started with the editing instruction Qw for the waveform W as a trigger. When the third editing process Sa3 is started, the third editing unit 35 edits the current waveform data Dw according to the editing instruction Qw (Sa301).

The information management unit 40 increases the waveform version number Vw by "1" (Sa302). Further, the information management unit 40 maintains the note sequence version number Vn at the current value Cn (Sa303), and also maintains the feature sequence version number Vf at the current value Cf (Sa304). Then, the information management unit 40 saves the third history data Hw [Vn, Vf, Vw] representing the editing instruction Qw this time in the history area as new version data (Sa305).

As understood from the above explanation, every time the waveform data Dw is edited according to the editing instruction Qw, the third history data Hw [Vn, Vf, Vw] corresponding to the edited waveform data Dw is saved in the history area. (Sa305), the waveform version number Vw is increased (Sa302) while the note string version number Vn and the feature column version number Vf are maintained (Sa303 and Sa304). In addition, step Sa303 and step Sa304 may be omitted.

The display control unit 20 displays the waveform W represented by the waveform data Dw edited by the third editing unit 35 in the editing area Ew (Sa306). In the third editing process Sa3, the note string N in the editing area En and the feature string F in the editing area Ef are not updated.

FIG. 7 is an explanatory diagram of the data structure in the history area of the storage device 12. A plurality of first history data Hn [Vn, Vf = 0, Vw = 0] (note string data Dn) corresponding to different versions of the note sequence N are stored in the history area. For each of the plurality of first history data Hn [Vn, Vf = 0, Vw = 0], a plurality of second history data Hf [Vn] corresponding to the feature sequence F of different versions under the common note sequence N. , Vf, Vw = 0] is stored in the history area. Further, for each of the plurality of second history data Hf [Vn, Vf, Vw = 0], a plurality of third history data Hw [Vn, Vn, corresponding to different versions of the waveform W under the common feature sequence F. Vf, Vw] is stored in the history area. As described above, the hierarchical relationship is established in which the note sequence N is located above the feature sequence F and the feature sequence F is located above the waveform W. When the feature column F is edited, the feature column version number Vf is increased, and the waveform version number Vw corresponding to the lower layer is set to "0" while the note string version number Vn corresponding to the upper layer is maintained. It is initialized.

8 to 10 are flowcharts illustrating a specific procedure of the management process Sb (Sb1, Sb2 and Sb3) that manages the version according to the instruction from the user. FIG. 8 is a flowchart of the first management process Sb1 regarding the version of the note string N. The first management process Sb1 is started with the instruction to change the note string version number Vn.

The numerical value of the note string version number Vn after the change according to the instruction from the user is referred to as "set value Xn" below. When the user directly changes the note string version number Vn in the operation area Gn, the changed numerical value (that is, the numerical value specified by the user) corresponds to the set value Xn. Further, when the user operates the operation image Gn1, the numerical value (= Cn-1) immediately before the current value Cn of the note string version number Vn corresponds to the set value Xn. On the other hand, when the user operates the operation image Gn2, the numerical value (= Cn + 1) immediately after the current value Cn of the note string version number Vn corresponds to the set value Xn.

When the first management process Sb1 is started, the information management unit 40 changes the note string version number Vn from the current value Cn to the set value Xn (Sb101).

The information management unit 40 sets the feature column version number Vf to the latest value Yf corresponding to the set value Xn of the note string N (Sb102). The latest value Yf is the number of the latest version among the plurality of versions of the feature string F generated for each edit instruction Qf under the note string N of the version corresponding to the set value Xn.

The information management unit 40 sets the waveform version number Vw to the latest value Yw corresponding to the set value Xn of the note string N (Sb103). The latest value Yw is the number of the latest version among a plurality of versions of the waveform W generated for each edit instruction Qw under the note string N of the version corresponding to the set value Xn.

The information management unit 40 has the first history data Hn [Vn = Xn, Vf = 0, Vw = 0] of the note sequence N and the second history data Hf [Vn = Xn, Vf = 1, Vw = = of the feature sequence F. 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] and the third history data Hw [Vn = Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf] of the waveform W. , Vw = Yw] is acquired from the history area of the storage device 12 (Sb104). The acquisition of the second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] is when the feature amount F is actually edited. Is executed, and is not executed when the feature amount F is not edited. The first history data Hn [Vn = Xn, Vf = 0, Vw = 0] of the note string N is the note string data Dn representing the version of the note string N in which the note string version number Vn is the set value Xn. In the second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] of the feature column F, the note sequence version number Vn is the set value Xn. It is data representing the time series of the edit instruction Qf before the Yfth among the one or more edit instruction Qf sequentially given by the user under the note string N. The third history data Hw [Vn = Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf, Vw = Yw] of the waveform W is the version in which the note string version number Vn is the set value Xn. Of the one or more edit instruction Qw sequentially given by the user under the note sequence N of the note sequence N and the feature sequence F of the version in which the feature column version number Vf is the latest value Yf, the edit instruction Qw before the Yw th It is data representing a time series.

The first generation unit 32 is characterized by supplying the first history data Hn [Vn = Xn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 to the first generation model M1. Generate column data Df (Sb105). The second editorial unit 33 has one or more second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] acquired by the information management unit 40. The feature column data Df is sequentially edited according to the editing instruction Qf represented by (Sb106). That is, the feature sequence data Df edited according to the edit instruction Qf up to the Yf th is generated under the note sequence N corresponding to the set value Xn. The editing by the second editing unit 33 is a small part of the feature sequence data Df over a plurality of notes. For example, only a very small part of the whole music, such as the attack part of a specific note in the music, or the first two notes in the third phrase in the music, is edited.

The second generation unit 34 is an input including the first history data Hn [Vn = Xn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 and the edited feature sequence data Df. The waveform data Dw is generated by supplying the data Din to the second generation model M2 (Sb107). The third editorial unit 35 has one or more third history data Hw [Vn = Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf, Vw = Yw] acquired by the information management unit 40. The waveform data Dw is sequentially edited according to the editing instruction Qw represented by (Sb108). That is, the waveform data Dw edited according to the edit instruction Qw up to the Ywth th is generated under the note string N corresponding to the set value Xn and the feature string F corresponding to the latest value Yf. If the second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] does not exist, the third history data Hw [Vn = Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf, Vw = Yw] are not acquired. That is, the waveform data Dw is not edited in step Sb108, and the waveform data Dw is determined as final data. When editing to move the waveform W in the direction of the time axis is instructed, for example, only the editing instruction Qw "move the section from time point 1 to time point 2 by X milliseconds" is the third history data Hw [Vn = It is saved as Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf, Vw = Yw]. Therefore, the amount of data stored in the storage device 12 can be significantly reduced as compared with the form in which the sample data of the waveform W after movement is stored over the entire music. The same applies to the editing of the volume or the editing of the filter for the waveform W. For editing the volume for the waveform W, the transition of the volume change in the section of the editing is saved, and for editing the filter for the waveform W, the parameters of the filter in the section of the editing are saved.

The display control unit 20 updates the edit screen G (Sb109-Sb111). Specifically, the display control unit 20 edits the note string N represented by the first history data Hn [Vn = Xn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40. It is displayed in En, and the display of the note string version number Vn in the operation area Gn is updated to the set value Xn (Sb109). That is, the note string N after editing by the Xnth editing instruction Qn is displayed in the editing area En.

Further, the display control unit 20 displays the feature column F represented by the feature column data Df edited by the second editing unit 33 in the edit area Ef, and displays the feature column version number Vf of the operation area Gf in the latest value Yf. Update (Sb110). That is, the feature column F corresponding to the set value Xn and the latest value Yf is displayed in the editing area E2. Similarly, the display control unit 20 displays the waveform W represented by the waveform data Dw edited by the third editing unit 35 in the editing area Ew, and updates the display of the waveform version number Vw in the operation area Gw to the latest value Yw. (Sb111). That is, the waveform W corresponding to the set value Xn, the latest value Yf, and the latest value Yw is displayed in the editing area Ew. In the above state, the user can give an editing instruction (Qn, Qf or Qw) for each of the note sequence N, the feature sequence F and the waveform W.

FIG. 9 is a flowchart of the second management process Sb2 regarding the version of the feature column F. The second management process Sb2 is started with the instruction to change the feature column version number Vf.

The numerical value of the feature column version number Vf after the change according to the instruction from the user is referred to as "set value Xf" below. When the user directly changes the feature column version number Vf in the operation area Gf, the changed numerical value (that is, the numerical value specified by the user) corresponds to the set value Xf. Further, when the user operates the operation image Gf1, the numerical value (= Cf-1) immediately before the current value Cf of the feature column version number Vf corresponds to the set value Xf. On the other hand, when the user operates the operation image Gf2, the numerical value (= Cf + 1) immediately after the current value Cf of the feature column version number Vf corresponds to the set value Xf.

When the second management process Sb2 is started, the information management unit 40 changes the feature column version number Vf from the current value Cf to the set value Xf (Sb201). Further, the information management unit 40 maintains the note string version number Vn at the current value Cn (Sb202), and changes the waveform version number Vw from the current value Cw to the latest value Yw (Sb203). The latest value Yw of the waveform version number Vw is the number of the latest version among the plurality of versions of the waveform W generated for each edit instruction Qw under the feature column F of the version corresponding to the set value Xf.

The information management unit 40 has the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note sequence N and the second history data Hf [Vn = Cn, Vf = 1, Vw = = of the feature sequence F. 0] to Hf [Vn = Cn, Vf = Xf, Vw = 0] and the third history data Hw [Vn = Cn, Vf = Xf, Vw = 1] to Hw [Vn = Xn, Vf = Xf] of the waveform W. , Vw = Yw] is acquired from the history area of the storage device 12 (Sb204). The first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note string N is the note string data Dn representing the note string N of the current version. The second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Xf, Vw = 0] of the feature column F is under the note string N of the current version. It is data representing the time series of the edit instruction Qf before the Xfth among the one or more edit instruction Qf given sequentially by the user. The third history data Hw [Vn = Cn, Vf = Xf, Vw = 1] to Hw [Vn = Xn, Vf = Xf, Vw = Yw] of the waveform W is the version in which the note string version number Vn is the current value Cn. Of the one or more edit instruction Qw sequentially given by the user under the note sequence N of the note sequence N and the feature sequence F of the version in which the feature column version number Vf is the set value Xf, the edit instruction Qw before the Ywth It is data representing a time series.

The first generation unit 32 is characterized by supplying the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 to the first generation model M1. Generate column data Df (Sb205). The second editorial unit 33 has one or more second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Xf, Vw = 0] acquired by the information management unit 40. The feature column data Df is sequentially edited according to the editing instruction Qf represented by (Sb206). That is, the feature sequence data Df edited according to the edit instruction Qf up to the Xf th is generated under the note sequence N corresponding to the current value Cn.

The second generation unit 34 is an input including the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 and the edited feature sequence data Df. The waveform data Dw is generated by supplying the data Din to the second generation model M2 (Sb207). The third editorial unit 35 has one or more third history data Hw [Vn = Cn, Vf = Xf, Vw = 1] to Hw [Vn = Xn, Vf = Xf, Vw = Yw] acquired by the information management unit 40. The waveform data Dw is sequentially edited according to the editing instruction Qw represented by (Sb208). That is, the waveform data Dw edited according to the edit instruction Qw up to the Ywth th is generated under the note string N corresponding to the current value Cn and the feature string F corresponding to the set value Xf.

The display control unit 20 updates the edit screen G (Sb209-Sb210). Specifically, the display control unit 20 displays the feature column F represented by the feature column data Df edited by the second editing unit 33 in the edit area Ef, and sets the display of the feature column version number Vf of the operation area Gf. Update to the value Xf (Sb209). That is, the feature column F corresponding to the current value Cn and the set value Xf is displayed in the editing area Ef. Further, the display control unit 20 displays the waveform W represented by the waveform data Dw edited by the third editing unit 35 in the editing area Ew, and updates the display of the waveform version number Vw in the operation area Gw to the latest value Yw ( Sb210). That is, the waveform W corresponding to the current value Cn, the set value Xf, and the latest value Yw is displayed in the editing area Ew. In the above state, the user can give an editing instruction (Qn, Qf or Qw) for each of the note sequence N, the feature sequence F and the waveform W.

FIG. 10 is a flowchart of the third management process Sb3 regarding the version of the waveform W. The third management process Sb3 is started with the instruction to change the waveform version number Vw.

The numerical value of the waveform version number Vw after the change according to the instruction from the user is referred to as "set value Xw" below. When the user directly changes the waveform version number Vw in the operation area Gw, the changed numerical value (that is, the numerical value specified by the user) corresponds to the set value Xw. Further, when the user operates the operation image Gw1, the numerical value (= Cw-1) immediately before the current value Cw of the waveform version number Vw corresponds to the set value Xw. On the other hand, when the user operates the operation image Gw2, the numerical value (= Cw + 1) immediately after the current value Cw of the waveform version number Vw corresponds to the set value Xw.

When the third management process Sb3 is started, the information management unit 40 changes the waveform version number Vw from the current value Cw to the set value Xw (Sb301). Further, the information management unit 40 maintains the note sequence version number Vn at the current value Cn (Sb302) and the feature sequence version number Vf at the current value Cf (Sb303).

The information management unit 40 has the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note sequence N and the second history data Hf [Vn = Cn, Vf = 1, Vw = = of the feature sequence F. 0] to Hf [Vn = Cn, Vf = Cf, Vw = 0] and the third history data Hw [Vn = Cn, Vf = Cf, Vw = 1] to Hw [Vn = Cn, Vf = Cf] of the waveform W. , Vw = Xw] is acquired from the history area of the storage device 12 (Sb304). The first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note string N is the note string data Dn representing the note string N of the current version. In the second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Cf, Vw = 0] of the feature column F, the note sequence version number Vn is the set value Xn. It is data representing the time series of the edit instruction Qf before the Cfth among one or more edit instruction Qf sequentially given by the user under the note string N. The third history data Hw [Vn = Cn, Vf = Cf, Vw = 1] to Hw [Vn = Cn, Vf = Cf, Vw = Xw] of the waveform W is the note string N of the current version and the current version. It is data representing the time series of the edit instruction Qw before the Xwth among one or more edit instruction Qw sequentially given by the user under the feature column F.

The first generation unit 32 is characterized by supplying the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 to the first generation model M1. Generate column data Df (Sb305). The second editorial unit 33 has one or more second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Cf, Vw = 0] acquired by the information management unit 40. The feature column data Df is sequentially edited according to the editing instruction Qf represented by (Sb306). That is, the feature sequence data Df edited according to the edit instruction Qf up to the Cf th is generated under the note sequence N corresponding to the current value Cn.

The second generation unit 34 is an input including the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] (note string data Dn) acquired by the information management unit 40 and the edited feature sequence data Df. The waveform data Dw is generated by supplying the data Din to the second generation model M2 (Sb307). The third editorial unit 35 has one or more third history data Hw [Vn = Cn, Vf = Cf, Vw = 1] to Hw [Vn = Cn, Vf = Cf, Vw = Xw] acquired by the information management unit 40. The waveform data Dw is sequentially edited according to the editing instruction Qw represented by (Sb308). That is, the waveform data Dw edited according to the edit instruction Qw up to the Xwth is generated under the note string N corresponding to the current value Cn and the feature string F corresponding to the current value Cf.

The display control unit 20 updates the edit screen G (Sb309). Specifically, the display control unit 20 displays the waveform W represented by the waveform data Dw edited by the third editing unit 35 in the editing area Ew, and displays the waveform version number Vw in the operation area Gw as the set value Xw. Update. That is, the waveform W corresponding to the current value Cn, the current value Cf, and the set value Xf is displayed in the editing area Ew.

As described above, in the first embodiment, the note sequence data Dn and the feature sequence data Df are edited according to the instructions (editing instruction Qn and editing instruction Qf) from the user. Therefore, it is possible to generate waveform data Dw that precisely reflects the instruction from the user, as compared with the configuration in which only the note string data Dn is edited in response to the instruction from the user.

Further, when the note string data Dn is edited, the note string version number Vn is increased, the numerical value of the feature string version number Vf is initialized, and when the feature string data Df is edited, the note is used. The numerical value of the feature column version number Vf is increased while the numerical value of the column version number Vn is maintained. Then, among the plurality of numerical values of the note string version number Vn, the first history data Hn [Vn, Vf, Vw] corresponding to the set value Xn according to the instruction from the user, and the plurality of numerical values of the feature column version number Vf. Of these, the waveform data Dw is generated by using at least one of the second history data Hf [Vn, Vf, Vw] corresponding to the set value Xf according to the instruction from the user. Therefore, the user can instruct the editing of the note sequence data Dn and the feature sequence data Df while generating the waveform data Dw by trial and error for different combinations of the note sequence version number Vn and the feature sequence version number Vf.

B: Second Embodiment The second embodiment will be described. For the elements whose functions are the same as those of the first embodiment in each of the embodiments exemplified below, the same reference numerals as those used in the description of the first embodiment are used, and detailed description of each is appropriately omitted. do.

FIG. 11 is a schematic diagram of the editing screen G in the second embodiment. In the editing screen G of the second embodiment, the operation image B2 is added to the same elements as those of the first embodiment. The operation image B2 is an image (specifically, a pull-down menu) for the user to select the pronunciation style of the synthetic sound. The user can select a desired pronunciation style from a plurality of pronunciation styles by operating the operation device 15.

Pronunciation style means a feature related to how to pronounce. For example, when the synthetic sound is a musical instrument sound, the pronunciation style is a characteristic of how the musical instrument is played. Further, for example, when the synthetic sound is a singing sound, the pronunciation style is a feature (sung around) regarding how to sing the music. Specifically, a suitable pronunciation method for each music genre, such as pop / rock / rap, is exemplified as a pronunciation style. In addition, the musical expression of playing or singing, such as bright / quiet / violent, is also exemplified as a pronunciation style.

FIG. 12 is a block diagram illustrating a functional configuration of the control device 11 in the second embodiment. The pronunciation style s selected by the user in the operation on the operation image B2 is instructed to the first generation unit 32 and the second generation unit 34 of the second embodiment.

The first generation unit 32 generates the feature sequence data Df from the note sequence data Dn and the pronunciation style s. The feature sequence data Df is time series data representing a time series of feature quantities (for example, fundamental frequency) related to a synthetic sound obtained by reproducing the note sequence N represented by the note sequence data Dn in the pronunciation style s.

Specifically, the first generation unit 32 generates the feature column data Df using the first generation model M1. The first generative model M1 is a statistical inference model that outputs feature sequence data Df by inputting note sequence data Dn and pronunciation style s. Similar to the first embodiment, the first generative model M1 is composed of a deep neural network having an arbitrary structure such as a convolutional neural network or a recurrent neural network. Specifically, the first generation model M1 includes a program that causes the control device 11 to execute an operation for generating feature sequence data Df from the note string data Dn and the pronunciation style s, and a plurality of variables applied to the operation. It is realized by the combination of.

A plurality of variables defining the first generation model M1 are set in advance by machine learning using a plurality of first training data and stored in the storage device 12. Each of the plurality of first training data includes the note sequence data Dn, the set of pronunciation styles s, and the feature sequence data Df (correct answer value). In the machine learning of the first generation model M1, the feature sequence data Df output by the provisional first generation model M1 for the note sequence data Dn and the pronunciation style s of each first training data, and the first training. The plurality of variables of the first generation model M1 are updated iteratively so that the error with the data feature column data Df is reduced. Therefore, the first generative model M1 outputs statistically valid feature sequence data Df for an unknown combination of note sequence data Dn and pronunciation style s under a tendency latent in a plurality of first training data. do.

The second generation unit 34 generates waveform data Dw from the note sequence data Dn, the feature sequence data Df, and the pronunciation style s. The waveform data Dw is time-series data representing the waveform of the synthetic sound sound obtained by pronouncing the note sequence N represented by the note sequence data Dn in the pronunciation style s.

Specifically, the second generation unit 34 generates the waveform data Dw using the second generation model M2. The second generative model M2 is a statistical inference model that outputs waveform data Dw by inputting note sequence data Dn, feature sequence data Df, and pronunciation style s. Similar to the first embodiment, the second generative model M2 is composed of a deep neural network having an arbitrary structure such as a convolutional neural network or a recurrent neural network. Specifically, the second generation model M2 is applied to a program that causes the control device 11 to execute an operation of generating waveform data Dw from the note string data Dn, the feature sequence data Df, and the pronunciation style s, and the operation. It is realized by combining with multiple variables.

A plurality of variables defining the second generation model M2 are set in advance by machine learning using a plurality of second training data and stored in the storage device 12. Each of the plurality of second training data includes a set of the note sequence data Dn, the feature sequence data Df, and the pronunciation style s, and the waveform data Dw (correct answer value). In the machine learning of the second generative model M2, the waveform data Dw output by the tentative second generative model M2 for the note sequence data Dn, the feature sequence data Df, and the pronunciation style s of each second training data, A plurality of variables of the second generation model M2 are iteratively updated so that the error of the second training data with the waveform data Dw is reduced. Therefore, the second generative model M2 has a statistically valid waveform for an unknown combination of the note sequence data Dn, the feature sequence data Df, and the pronunciation style s under the tendency latent in the plurality of second training data. Output data Dw.

In step Sa201 of the second editing process Sa2, the first editing unit 31 obtains the feature sequence data Df representing the feature sequence F of the synthetic sound in which the note sequence N is pronounced by the pronunciation style s selected by the user. Edit according to the edit instruction Qf. Further, in the step Sa205 of the second editing process Sa2, the information management unit 40 stores the second history data Hf [Vn, Vf, Vw] corresponding to the edited feature column data Df for each version of the feature column data Df. It is saved in the history area of the device 12.

As understood from the above explanation, the feature sequence data Df corresponding to the pronunciation style s and the waveform data Dw corresponding to the pronunciation style s are generated under the specific note sequence N. On the other hand, the note sequence N is not affected by the pronunciation style s. Therefore, as illustrated in FIG. 13, for the first history data Hn [Vn, Vf, Vw] (note string data Dn) corresponding to one note sequence N, different feature sequences F for each pronunciation style s. A plurality of second history data Hf [Vn, Vf, Vw] corresponding to the above and a plurality of third history data Hw [Vn, Vf, Vw] corresponding to different waveforms W are stored in the history area of the storage device 12. It will be saved.

Next, a specific example of the operation in the second embodiment will be described. In the first editing process Sa1, the feature sequence data Df representing the feature sequence F of the synthetic sound that pronounces the note sequence N in the pronunciation style s is generated by the first processing unit (Sa106), and represents the waveform W of the synthetic sound. Waveform data Dw is generated by the second processing unit (Sa107).

In the second editing process Sa2, the second editing unit 33 edits the feature sequence data Df according to the pronunciation style s according to the editing instruction Qf from the user. The information management unit 40 history the second history data Hf [Vn, Vf, Vw] corresponding to the edited feature column data Df for each edit of the feature column data Df (that is, for each version of the feature column data Df). Save to area.

Similarly, in the third editing process Sa3, the third editing unit 35 edits the waveform data Dw according to the pronunciation style s according to the editing instruction Qw from the user. The information management unit 40 saves the third history data Hw [Vn, Vf, Vw] corresponding to the edited waveform data Dw in the history area for each edit of the waveform data Dw (that is, for each version of the waveform data Dw). do.

In the second embodiment, in the state where the pronunciation style s is selected, the first management process Sb1 is started with the instruction to change the note string version number Vn. In step Sb104 of the first management process Sb1, the information management unit 40 has the first history data Hn [Vn = Xn, Vf = 0, Vw = 0] of the note sequence N and the feature sequence F corresponding to the pronunciation style s. The second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, Vf = Yf, Vw = 0] and the third history data Hw of the waveform W corresponding to the pronunciation style s. [Vn = Xn, Vf = Yf, Vw = 1] to Hw [Vn = Xn, Vf = Yf, Vw = Yw] are acquired from the history area. In steps Sb105 to Sb108 of the first management process Sb1, the feature sequence data Df of the feature sequence F corresponding to the pronunciation style s and the waveform data Dw of the waveform W corresponding to the pronunciation style s are generated.

In the second embodiment, in the state where the pronunciation style s is selected, the second management process Sb2 is started with the instruction to change the feature column version number Vf. In step Sb204 of the second management process Sb2, the information management unit 40 has the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note sequence N and the feature sequence F corresponding to the pronunciation style s. The second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Xf, Vw = 0] and the third history data Hw of the waveform W corresponding to the pronunciation style s. [Vn = Cn, Vf = Xf, Vw = 1] to Hw [Vn = Xn, Vf = Xf, Vw = Yw] are acquired from the history area. Specifically, the "feature column F corresponding to the pronunciation style s" is a feature corresponding to the note sequence version number Vn (set value Xn), the pronunciation style s, and the feature sequence version number Vf (latest value Yf). Column F. Specifically, the "waveform W corresponding to the pronunciation style s" includes a note string version number Vn (set value Xn), a pronunciation style s, a feature string version number Vf (latest value Yf), and a waveform version. It is a waveform W corresponding to the number Vw (latest value Yw). In steps Sb205 to Sb208 of the second management process Sb2, the feature sequence data Df of the feature sequence F corresponding to the pronunciation style s and the waveform data Dw of the waveform W corresponding to the pronunciation style s are generated. Specifically, the "feature sequence F corresponding to the pronunciation style s" is a feature corresponding to the note sequence version number Vn (current value Cn), the pronunciation style s, and the feature sequence version number Vf (set value Xf). Column F. Specifically, the "waveform W corresponding to the pronunciation style s" includes a note string version number Vn (current value Cn), a pronunciation style s, a feature string version number Vf (set value Xf), and a waveform version. It is a waveform W corresponding to the number Vw (latest value Yw).

In the second embodiment, in the state where the pronunciation style s is selected, the third management process Sb3 is started with the instruction to change the waveform version number Vw. In step Sb304 of the third management process Sb3, the information management unit 40 has the first history data Hn [Vn = Cn, Vf = 0, Vw = 0] of the note sequence N and the feature sequence F corresponding to the pronunciation style s. The second history data Hf [Vn = Cn, Vf = 1, Vw = 0] to Hf [Vn = Cn, Vf = Cf, Vw = 0] and the third history data Hw of the waveform W corresponding to the pronunciation style s. [Vn = Cn, Vf = Cf, Vw = 1] to Hw [Vn = Cn, Vf = Cf, Vw = Xw] are acquired from the history area. In steps Sb305 to Sb308 of the third management process Sb3, the feature sequence data Df of the feature sequence F corresponding to the pronunciation style s and the waveform data Dw of the waveform W corresponding to the pronunciation style s are generated. Specifically, the "feature sequence F corresponding to the pronunciation style s" is a feature corresponding to the note sequence version number Vn (current value Cn), the pronunciation style s, and the feature sequence version number Vf (current value Cf). Column F. Further, the "waveform W corresponding to the pronunciation style s" specifically includes a note string version number Vn (current value Cn), a pronunciation style s, a feature string version number Vf (current value Cf), and a waveform version. It is a waveform W corresponding to the number Vw (set value Xw).

Here, pay attention to the pronunciation style s1 and the pronunciation style s2 that the user can select from a plurality of pronunciation styles s. The pronunciation style s1 and the pronunciation style s2 are different pronunciation styles s. The pronunciation style s1 is an example of the "first pronunciation style", and the pronunciation style s2 is an example of the "second pronunciation style".

First, assume that the pronunciation style s1 is selected. In the second editing process Sa2, the second editing unit 33 edits the feature sequence data Df corresponding to the pronunciation style s1 according to the editing instruction Qf from the user. Then, each time the feature column data Df is edited, the information management unit 40 saves the second history data Hf [Vn, Vf, Vw] corresponding to the edited feature column data Df in the history area. Similarly, in the third editing process Sa3, the third editing unit 35 edits the waveform data Dw according to the pronunciation style s1 according to the editing instruction Qw from the user. Then, each time the waveform data Dw is edited, the information management unit 40 saves the third history data Hw [Vn, Vf, Vw] corresponding to the edited waveform data Dw in the history area. The feature sequence data Df or waveform data Dw generated when the pronunciation style s1 is selected is an example of "first time series data". Further, the editing instruction Qf or the editing instruction Qw given by the user with the pronunciation style s1 selected is an example of the "first instruction".

When the pronunciation style s1 is selected, the feature sequence corresponding to the pronunciation style s1 in step Sb104 of the first management process Sb1, step Sb204 of the second management process Sb2, and step Sb304 of the third management process Sb3. The feature sequence data Df of F and the waveform data Dw of the waveform W corresponding to the pronunciation style s1 are generated. That is, the feature sequence data Df and the waveform data corresponding to the history data H corresponding to the instruction (Xn, Xf, Xw) from the user among the plurality of history data H (Hn, Hf, Hw) corresponding to the pronunciation style s1. Dw is generated.

Next, assume that the pronunciation style s2 is selected. In the second editing process Sa2, the second editing unit 33 edits the feature sequence data Df corresponding to the pronunciation style s2 according to the editing instruction Qf from the user. Then, each time the feature column data Df is edited, the information management unit 40 saves the second history data Hf [Vn, Vf, Vw] corresponding to the edited feature column data Df in the history area. Similarly, in the third editing process Sa3, the third editing unit 35 edits the waveform data Dw corresponding to the pronunciation style s2 according to the editing instruction Qw from the user. Then, each time the waveform data Dw is edited, the information management unit 40 saves the third history data Hw [Vn, Vf, Vw] corresponding to the edited waveform data Dw in the history area. The feature sequence data Df or waveform data Dw generated when the pronunciation style s2 is selected is an example of "second time series data". Further, the editing instruction Qf or the editing instruction Qw given by the user with the pronunciation style s2 selected is an example of the "second instruction".

When the pronunciation style s2 is selected, the feature sequence corresponding to the pronunciation style s2 in the step Sb104 of the first management process Sb1, the step Sb204 of the second management process Sb2, and the step Sb304 of the third management process Sb3. The feature sequence data Df of F and the waveform data Dw of the waveform W corresponding to the pronunciation style s2 are generated. That is, the feature sequence data Df and the waveform data corresponding to the history data H corresponding to the instruction (Xn, Xf or Xw) from the user among the plurality of history data H (Hn, Hf and Hw) corresponding to the pronunciation style s2. Dw is generated.

As can be understood from the above examples, the editing processing unit 30 in the second embodiment has the feature sequence data Df and the waveform data Dw corresponding to the pronunciation style s1, or the feature sequence data Df and the waveform data corresponding to the pronunciation style s2. Dw is acquired according to the common version of the note string data Dn.

As illustrated above, in the second embodiment, the editing history of the feature sequence data Df and the waveform data Dw corresponding to the pronunciation style s1 is stored in the storage device 12, and the feature sequence data Df and the feature sequence data Df corresponding to the pronunciation style s2 are stored. The editing history of the waveform data Dw is stored in the storage device 12. Therefore, the editing of the feature sequence data Df or the waveform data Dw corresponding to the pronunciation style s1 and the editing of the feature sequence data Df or the waveform data Dw corresponding to the pronunciation style s2 are performed by trial and error according to the instruction from the user. It is possible to execute.

For example, when the user instructs the comparison between the pronunciation styles s by the operation of the operation device 15, the display control unit 20 causes the display device 14 to display the comparison screen U of FIG. The comparison screen U includes a first region U1, an operation image U1a (call), an operation image U1b (reproduction), a second region U2, an operation image U2a (call), and an operation image U2b (reproduction).

In each of the first region U1 and the second region U2, the first history data Hn [Vn, Vf, Vw], the second history data Hf [Vn, Vf, Vw] and the third history data Hw [Vn, Vf, The hierarchical relationship with Vw] is displayed. By operating the operating device 15, the user can select desired historical data H for each of the first region U1 and the second region U2. Specifically, the user selects the desired history data H for each of the first region U1 and the second region U2 by designating the pronunciation style s and each version number (Vn, Vf, Vw). ..

When the user selects the operation image U1a (call), the control device 11 acquires the history data H selected in the first area U1 from the storage device 12, and displays the edit screen G corresponding to the history data H. Display on the display device 14. Specifically, the control device 11 determines the first history data Hn of the note sequence N according to the pronunciation style s of the history data H selected for the first region U1 and each version number (Vn, Vf, Vw). [Vn = Xn, Vf = 0, Vw = 0] and the second history data Hf [Vn = Xn, Vf = 1, Vw = 0] to Hf [Vn = Xn, of the feature column F corresponding to the pronunciation style s. Vf = Xf, Vw = 0] and the third history data Hw [Vn = Xn, Vf = Xf, Vw = 1] to Hw [Vn = Xn, Vf = Xf, Vw =] of the waveform W corresponding to the pronunciation style s. Xw] and get from the history area. The control device 11 uses each history data H acquired from the history area to display the feature sequence data Df of the feature sequence F and the waveform data Dw of the waveform W corresponding to the version numbers (Vn, Vf, Vw) of the pronunciation style s. And generate. Then, the control device 11 has a note sequence indicated by the first history data Hn [Vn = Xn, Vf = 0, Vw = 0], a feature sequence F indicated by the feature sequence data Df, and a waveform W indicated by the waveform data Dw. The display screen G including the above is displayed on the display device 14. When the user selects the operation image U1b (reproduction), the control device 11 supplies the sound emitting device 13 with the acoustic signal Z corresponding to the waveform data Dw generated in the above procedure for the first region U1. By doing so, the synthetic sound is reproduced.

Similarly, when the user selects the operation image U2a (call), the control device 11 acquires the history data H selected in the second region U2 from the storage device 12, and edits the history data H according to the history data H. The screen G is displayed on the display device 14. Specifically, the control device 11 sets the pronunciation style s and each version number (Vn, Vf, Vw) specified by the user for the second region U2 by the same procedure as described above for the first region U1. Generate the corresponding feature sequence data Df and waveform data Dw. Then, the control device 11 has a note sequence indicated by the first history data Hn [Vn = Xn, Vf = 0, Vw = 0], a feature sequence F indicated by the feature sequence data Df, and a waveform W indicated by the waveform data Dw. The display screen G including the above is displayed on the display device 14. When the user selects the operation image U2b (reproduction), the control device 11 supplies the sound emitting device 13 with the acoustic signal Z corresponding to the waveform data Dw generated in the above procedure for the second region U2. By doing so, the synthetic sound is reproduced.

As can be understood from the above examples, the user mutually compares the combination of the version and the pronunciation style s selected from the first region U1 with the combination of the version and the pronunciation style s selected from the second region U2. While, it is possible to adjust the note sequence N, the feature sequence F, the waveform W, and the pronunciation style s.

C: Third Embodiment FIG. 15 is an explanatory diagram of the synthetic sound in the third embodiment. The synthetic sound of the third embodiment is composed of a plurality of tracks T (T1, T2, ...) Parallel to each other on the time axis. For example, when a musical instrument sound composed of a plurality of performance parts is regarded as a synthetic sound, each performance part corresponds to the track T. Further, when a singing sound composed of a plurality of singing parts is used as a synthetic sound, each singing part corresponds to the track T.

Each of the plurality of tracks T includes a plurality of sections (hereinafter referred to as "unit intervals") R that do not overlap each other on the time axis. Each of the plurality of unit intervals R is an interval (region) including the note string N on the time axis. That is, a unit interval R is set for each note sequence N, with a set of a plurality of notes that are close to each other on the time axis as a note sequence N. The time length of each unit interval R is a variable length according to the total number of notes in the note sequence N, the continuation length of each note, and the like.

FIG. 16 is a schematic diagram of the editing screen G in the third embodiment. Information on one unit interval R selected by the user (note sequence N, feature sequence F, or waveform) among the plurality of unit intervals R of one track T selected by the user from the plurality of tracks T of the synthetic sound. W) is displayed on the edit screen G. In the editing screen G of the second embodiment, the operation area Gt and the operation area Gr are added to the same elements as those of the first embodiment.

The operation area Gt is an area related to the track T of the synthetic sound. Specifically, the track version number Vt, the operation image Gt1 and the operation image Gt2 are displayed in the operation area Gt. The track version number Vt is a number representing the version of the track T displayed on the edit screen G. The track version number Vt is incremented by 1 each time the information about the track T displayed on the edit screen G (note string N, feature column F, or waveform W) is edited. Further, the user can change the track version number Vt in the operation area Gt to an arbitrary numerical value by operating the operation device 15.

The operation image Gt1 and the operation image Gt2 are software buttons that can be operated by the user using the operation device 15. The operation image Gt1 is an operator for instructing the user to return the information (note string N, feature sequence F, or waveform W) related to the track T to the state before the execution of the immediately preceding edit (Undo). Further, the operation image Gt2 is an operator for instructing the user to perform the editing canceled by the operation on the operation image Gt1 again (Redo).

The operation area Gr is an area related to the unit interval R of the synthetic sound. Specifically, the section version number Vr, the operation image Gr1 and the operation image Gr2 are displayed in the operation area Gr. The section version number Vr is a number representing the version of the unit section R displayed on the edit screen G. The section version number Vr is incremented by 1 each time the information regarding the unit interval R displayed on the edit screen G (note sequence N, feature sequence F, or waveform W) is edited. Further, the user can change the track version number Vt in the operation area Gt to an arbitrary numerical value by operating the operation device 15.

The operation image Gr1 and the operation image Gr2 are software buttons that can be operated by the user using the operation device 15. The operation image Gr1 is an operator for instructing the user to return the information (note string N, feature sequence F, or waveform W) regarding the unit interval R to the state before the execution of the immediately preceding edit (Undo). Further, the operation image Gr2 is an operator for instructing the user to execute (Redo) the editing canceled by the operation on the operation image Gr1 again.

The editing process Sa (Sa1-Sa3) or the management process Sb (Sb1-Sb3) is executed for each of the plurality of unit intervals R in one track T displayed on the editing screen G. In the editing process Sa, each time any one of the note sequence N, the feature sequence F, and the waveform W is edited, the information management unit 40 increases the track version number Vt and the section version number Vr by one. Further, when the user operates the operation image (Gn1, Gf1, Gw1, Gn2, Gf2 or Gw2), the information management unit 40 similarly increases the track version number Vt and the section version number Vr by one.

The same effect as that of the first embodiment is realized in the third embodiment. Further, in the third embodiment, the user generates the waveform data Dw by trial and error for each of the plurality of unit intervals R on the time axis, while the note sequence data Dn, the feature sequence data Df, and the waveform data Dw. You can instruct each edit with.

D: Modification example The specific modification mode added to each of the above-exemplified embodiments will be exemplified below. Two or more embodiments arbitrarily selected from the following examples may be appropriately merged to the extent that they do not contradict each other.

(1) In each of the above-mentioned forms, the note string data Dn of each version is stored in the history area as the first history data Hn [Vn, Vf, Vw], but the first history data Hn [Vn, Vf, Vw] And the format of the first history data Hn [Vn, Vf, Vw] are not limited to the above examples. For example, the first history data Hn [Vn, Vf, Vw] indicating how the note string data Dn is edited (that is, the time series of the edit instruction Qn) may be saved. As understood from the above description, the first history data Hn [Vn, Vf, Vw] is comprehensively expressed as data corresponding to the edited note sequence N.

(2) In each of the above-mentioned forms, the second history data Hf [Vn, Vf, Vw] indicating how the feature column data Df is edited (that is, the time series of the edit instruction Qf) is stored in the history area. However, the matters represented by the second history data Hf [Vn, Vf, Vw] and the format of the second history data Hf [Vn, Vf, Vw] are not limited to the above examples. For example, the feature column data Df after editing according to the editing instruction Qf may be saved in the history area as the second history data Hf [Vn, Vf, Vw]. As can be understood from the above examples, the second history data Hf [Vn, Vf, Vw] is comprehensively represented as data corresponding to the edited feature column data Df.

(3) In each of the above-mentioned forms, the third history data Hw [Vn, Vf, Vw] indicating how the waveform data Dw is edited (that is, the time series of the edit instruction Qw) is saved in the history area. , The matters represented by the third history data Hw [Vn, Vf, Vw] and the format of the third history data Hw [Vn, Vf, Vw] are not limited to the above examples. For example, the waveform data Dw after editing according to the editing instruction Qw may be saved in the history area as the third history data Hw [Vn, Vf, Vw]. As can be understood from the above examples, the third history data Hw [Vn, Vf, Vw] is comprehensively expressed as data corresponding to the edited waveform data Dw.

(4) In each of the above-described forms, the feature sequence F having the fundamental frequency of the synthesized sound as the feature quantity is illustrated, but the feature quantity represented by the feature sequence data Df is not limited to the fundamental frequency. For example, the frequency spectrum of the synthesized sound in the frequency domain (for example, the intensity spectrum) or the time-series data representing the time series (feature sequence F) of the feature amount with the sound pressure level on the time axis as the feature sequence data. It may be Df. The feature sequence data Df is comprehensively represented as time series data representing a time series (feature sequence F) of the feature amount of the note sequence data Dn.

(5) In each of the above-described embodiments, the second generation unit 34 generates the waveform data Dw from the note sequence data Dn and the feature sequence data Df, but the second generation unit 34 generates the waveform data Dw from the note sequence data Dn. Or a configuration in which the second generation unit 34 generates waveform data Dw from the feature column data Df is also assumed. That is, the second generation unit 34 is specified as an element that generates waveform data Dw from at least one of the note string data Dn and the waveform data Dw.

(6) In the second embodiment, the first generation model M1 that outputs the feature sequence data Df for the input including the pronunciation style s is exemplified, but the feature sequence data Df corresponding to the pronunciation style s is first generated. The configuration for the unit 32 to be generated is not limited to the above examples. For example, the feature sequence data Df may be generated by selectively using a plurality of first generation models M1 corresponding to different pronunciation styles s. The first generation model M1 corresponding to each pronunciation style s is constructed by machine learning using a plurality of first training data prepared for the pronunciation style s. The first generation unit 32 generates the feature sequence data Df by inputting the note sequence data Dn into the first generation model M1 corresponding to the pronunciation style s selected by the user among the plurality of first generation models M1. ..

Further, in the second embodiment, the second generation model M2 that outputs the waveform data Dw to the input including the pronunciation style s is exemplified, but the second generation unit 34 generates the waveform data Dw according to the pronunciation style s. The configuration for generation is not limited to the above examples. For example, the waveform data Dw may be generated by selectively using a plurality of second generation models M2 corresponding to different pronunciation styles s. The second generative model M2 corresponding to each pronunciation style s is constructed by machine learning using a plurality of second training data prepared for the pronunciation style s. The second generation unit 34 inputs the note sequence data Dn and the feature sequence data Df (input data Din) to the second generation model M2 corresponding to the pronunciation style s selected by the user among the plurality of second generation models M2. As a result, waveform data Dw is generated.

(7) In each of the above-described embodiments, the waveform W of the acoustic signal Z is displayed in the edit area Ew of the edit screen G, but the time series (that is, spectrogram) of the frequency spectrum of the acoustic signal Z is displayed on the edit screen G together with the waveform W. It may be displayed. For example, the editing screen G illustrated in FIG. 17 includes an editing area Ew1 and an editing area Ew2. In the editing area Ew1, the waveform W is displayed in the same manner as the editing area Ew in each of the above-described forms. On the other hand, in the editing area Ew2, the time series of the frequency spectrum of the acoustic signal Z is displayed. In addition to the editing instruction Qw for the waveform in the editing area Ew1, the user can give the editing instruction Qw for the frequency spectrum in the editing area Ew2 by operating the operation device 15.

(8) Note string data Dn is time-series data representing a note sequence N having a plurality of notes on the time axis as elements. The feature sequence data Df is time-series data representing the feature sequence F having a plurality of feature quantities on the time axis as elements. The waveform data Dw is time-series data representing a waveform W having a plurality of samples on the time axis as elements. As understood from the above examples, the note sequence data Dn, the feature sequence data Df, and the waveform data Dw are comprehensively represented as time series data representing a time series of a plurality of elements.

(9) In each of the above-described embodiments, the deep neural network is exemplified as the first generation model M1 and the second generation model M2, but the configurations of the first generation model M1 and the second generation model M2 are arbitrary. For example, a statistical inference model of another structure such as HMM (Hidden Markov Model) may be used as the first generation model M1 or the second generation model M2.

(10) In each of the above-mentioned forms, the synthesis of the synthetic sound corresponding to the note string N is illustrated, but each of the above-mentioned forms can be used in any scene for processing time-series data representing a time-series of a plurality of elements. Will be done. For example, in each of the above-mentioned forms, the upper layer corresponds to the note sequence N, the middle layer corresponds to the feature sequence F, and the lower layer corresponds to the waveform W. Each layer in the scene is a combination illustrated below.

For example, in the scene of a self-operated song that generates a melody, the note strings constituting the melody correspond to the upper layer, the time series of the chords in the melody corresponds to the middle layer, and the accompaniment sound that harmonizes with the melody. The note sequence corresponds to the lower layer. Further, in the voice synthesis scene in which the voice corresponding to the character string is synthesized, the character string corresponds to the upper layer, the pronunciation style of the voice corresponds to the middle layer, and the waveform of the voice corresponds to the lower layer. do. In the signal processing scene where various signals are processed, the waveform of the signal corresponds to the upper layer, the time series of the feature amount of the signal corresponds to the middle layer, and the time series of the parameters related to the processing for the signal corresponds to the upper layer. Corresponds to the lower layer. In any of the above-exemplified forms, the upper layer data is expressed as "upper data", the middle layer data is expressed as "middle data", and the lower layer data is expressed as "lower data". To. The lower-level data is data representing the content actually used by the user (for example, the waveform W in each of the above-mentioned forms).

Note that each note constituting the note string N in each of the above-mentioned forms and each character constituting the character string in speech synthesis are comprehensively expressed as symbols indicating sounds. Further, the note string N and the character string are comprehensively represented as a symbol string in which a plurality of symbols are arranged in time series.

(11) As described above, the functions of the acoustic processing system exemplified above are realized by the cooperation of the single or a plurality of processors constituting the control device 11 and the program stored in the storage device 12. The program according to the present disclosure may be provided and installed in a computer in a form stored in a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disc) such as a CD-ROM is a good example, but a semiconductor recording medium, a magnetic recording medium, or the like is known as arbitrary. Recording media in the form of are also included. The non-transient recording medium includes any recording medium other than the transient propagation signal (transitory, propagating signal), and the volatile recording medium is not excluded. Further, in the configuration in which the distribution device distributes the program via the communication network, the storage device 12 that stores the program in the distribution device corresponds to the above-mentioned non-transient recording medium.

E: Addendum For example, the following configuration can be grasped from the above-exemplified forms.

In the information processing method according to one aspect (aspect 1) of the present disclosure, the first time-series data representing the time-series of the feature amount of the sound in which the symbol string is sounded in the first pronunciation style is given as the first instruction from the user. The first history data corresponding to the edited first time-series data is saved as new version data for each edit of the first time-series data, and what is the first pronunciation style? The second time-series data representing the time series of the feature amount of the sound that pronounced the symbol string with a different second pronunciation style is edited according to the second instruction from the user, and the second time-series data is edited. For each time, the second history data corresponding to the edited second time-series data is saved as new version data, and among the saved first history data of different versions, from the user. Corresponds to the first time-series data corresponding to the first history data corresponding to the instruction, or the second history data corresponding to the instruction from the user among the plurality of saved second history data of different versions. 2nd time series data to be acquired.

According to the above aspect, the history of editing the first time-series data corresponding to the first pronunciation style is saved, and the history of editing the second time-series data corresponding to the second pronunciation style is saved. Therefore, the editing of the first time-series data corresponding to the first pronunciation style and the editing of the second time-series data corresponding to the second pronunciation style are executed by trial and error according to the instruction from the user. Is possible. The "symbol string" is, for example, a musical note string or a character string.

In the specific example of the first aspect (the second aspect), the symbol string is a note sequence including a plurality of notes arranged in a time series. Further, in the specific example of the second aspect (aspect 3), the note sequence data representing the note sequence is edited according to the instruction from the user, and the first time series data and the second time series data are common. Generated from the note string data of the version of.

In the specific example of the first to third aspects (aspect 4), in the acquisition, the first history data after the editing immediately before the plurality of first history data and the immediately preceding of the plurality of second history data. Acquire any of the second history data after editing. According to the above configuration, the first history data or the second history data before the execution of the immediately preceding edit (that is, the state in which the edit is canceled) can be acquired.

In the specific example of the first to third aspects (aspect 5), in the acquisition, the first history data of the version designated by the user among the plurality of first history data, and the plurality of second history data. Of these, any of the second history data of the version specified by the user is acquired. According to the above configuration, it is possible to acquire the first history data or the second history data corresponding to any version according to the instruction from the user.

The information processing system according to one aspect of the present disclosure edits the first time-series data representing the time-series of the feature amount of the sound that pronounces the symbol string in the first pronunciation style according to the first instruction from the user. Then, the second time series data representing the time series of the feature amount of the sound that pronounced the symbol string in the second pronunciation style different from the first pronunciation style is edited according to the second instruction from the user. Each time the editing processing unit edits the first time-series data, the first history data corresponding to the edited first time-series data is saved as new version data, and the second time-series data is edited. Each time, it is provided with an information management unit that saves the second history data corresponding to the edited second time-series data as new version data, and the information management unit has the saved different versions. The use of the first time-series data corresponding to the first history data according to the instruction from the user among the plurality of first history data, or the second history data of a plurality of different versions of the saved data. Acquire the second time-series data corresponding to the second history data according to the instruction from the person. The program according to one aspect of the present disclosure causes the computer system to function as the above information processing system.

100 ... Information processing system, 11 ... Control device, 12 ... Storage device, 13 ... Sound emitting device, 14 ... Display device, 15 ... Operation device, 20 ... Display control unit, 30 ... Editing processing unit, 31 ... First editing unit , 32 ... 1st generation unit, 33 ... 2nd editorial unit, 34 ... 2nd generation unit, 35 ... 3rd editorial unit, M1 ... 1st generation model, M2 ... 2nd generation model.

Claims

The first time-series data representing the time-series of the feature amount of the sound that pronounced the symbol string in the first pronunciation style is edited according to the first instruction from the user.
Every time the first time-series data is edited, the first history data corresponding to the edited first time-series data is saved as new version data.
The second time-series data representing the time-series of the feature amount of the sound that pronounced the symbol string in the second pronunciation style different from the first pronunciation style is edited according to the second instruction from the user.
Every time the second time-series data is edited, the second history data corresponding to the edited second time-series data is saved as new version data.
Among the plurality of saved different versions of the first history data, the first time-series data corresponding to the first history data in response to the instruction from the user, or the plurality of saved different versions of the first history data. An information processing method realized by a computer system that acquires second time-series data corresponding to the second history data according to an instruction from the user among the second history data.
The information processing method according to claim 1, wherein the symbol sequence is a musical note sequence including a plurality of musical notes arranged in a time series.
The note string data representing the note string is edited according to the instruction from the user, and the note string data is edited.
The information processing method according to claim 2, wherein the first time-series data and the second time-series data are generated from a common version of the note string data.
In the acquisition, either the first history data after the previous editing among the plurality of first history data and the second history data after the previous editing among the plurality of second history data are acquired. The information processing method according to any one of claims 1 to 3.
In the acquisition, the first history data of the version specified by the user among the plurality of first history data and the second history data of the version designated by the user among the plurality of second history data. The information processing method according to any one of claims 1 to 3.
The first time-series data representing the time-series of the feature amount of the sound in which the symbol string is pronounced in the first pronunciation style is edited according to the first instruction from the user, and the second pronunciation is different from the first pronunciation style. An editing processing unit that edits the second time-series data representing the time-series of the feature amount of the sound that pronounces the symbol string in the style according to the second instruction from the user.
Each time the first time-series data is edited, the first history data corresponding to the edited first time-series data is saved as new version data, and each time the second time-series data is edited, the edit is performed. It is equipped with an information management unit that saves the second history data corresponding to the second time series data later as new version data.
The information management unit is the first time-series data corresponding to the first history data according to the instruction from the user among the plurality of first history data of different versions saved, or the saved first history data. An information processing system that acquires second time-series data corresponding to the second history data in response to an instruction from the user among a plurality of second history data of different versions.
The first time-series data representing the time-series of the feature amount of the sound in which the symbol string is pronounced in the first pronunciation style is edited according to the first instruction from the user, and the second pronunciation is different from the first pronunciation style. An editing processing unit that edits the second time-series data representing the time-series of the feature amount of the sound that pronounces the symbol string in the style according to the second instruction from the user, and
Each time the first time-series data is edited, the first history data corresponding to the edited first time-series data is saved as new version data, and each time the second time-series data is edited, the edit is performed. Information management unit that saves the second history data corresponding to the second time series data later as new version data,
It is a program that makes a computer system function as
The information management unit is the first time-series data corresponding to the first history data according to the instruction from the user among the plurality of first history data of different versions saved, or the saved first history data. A program for acquiring second time-series data corresponding to the second history data in response to an instruction from the user among a plurality of second history data of different versions.