JP6003195B2 - Apparatus and program for performing singing synthesis - Google Patents

Description

  The present invention relates to a song synthesis technique, and more particularly to an apparatus and program suitable for song synthesis for rap.

  In the singing voice synthesizing apparatus, the singing voice is synthesized based on information indicating individual notes constituting the song and information indicating lyrics to be sung in accordance with the notes (for example, Patent Document 1). Therefore, in order for the synthesizer to perform singing synthesis, it is necessary to specify the lyrics to be sung in accordance with the notes for each note constituting the song.

JP 2011-128186 A

  By the way, in recent years, a singing method called rap has become popular. This rap is a singing method that sings in a rhythmic manner without melody but with rhymes at the end of a measure. Conventionally, a singing voice synthesis apparatus suitable for singing voice synthesis for such a lap has not been provided. If a conventional synthesizer is to synthesize a rap song, it is necessary to associate lyrics containing a relatively large number of phoneme strings with one note, and the operation for that is troublesome.

  This invention is made | formed in view of the above situations, and it aims at providing the song synthesis apparatus suitable for the song synthesis | combination of a rap.

  The present invention includes a phrase data generating unit that divides the lyrics indicated by the lyrics data into a plurality of fragments, generates a plurality of phrase data each indicating the utterance mode of the phrase of the synthesized singing sound from each fragment, and writes the phrase data to the storage unit; A singing voice synthesizing apparatus comprising: phrase reproducing means for reading out the phrase data from the storage means and generating a synthesized singing sound using the read phrase data in response to a reproduction instruction for phrase data. I will provide a.

  According to this invention, the lyrics are divided into a plurality of fragments by the phrase data generation means, and a plurality of phrase data indicating the utterance mode of the phrase of the synthesized singing sound is generated from each fragment and stored in the storage means. And when the reproduction | regeneration instruction | indication which makes desired phrase data object generate | occur | produces, the said phrase data is read from a memory | storage means by a phrase reproduction | regeneration means, and a synthetic | combination song sound is produced | generated using this phrase data. Therefore, it is possible to easily generate phrase data as a material for wrapping and easily wrap using the phrase data.

  The present invention can also be realized as a sequence data editing apparatus that generates sequence data for wrapping. This sequence editing apparatus is configured to display an operation means, a display means, and a sequence data editing area having a time axis on the display means, and a block indicating a phrase that is a synthesized singing sound of a fragment of lyrics according to the operation of the operation means Are laid out in the sequence data editing area, and the playback timing of the phrase corresponding to the block is determined based on the position of the block in the sequence data editing area in the time axis direction. Sequence data generating means for generating sequence data in which a phrase playback instruction corresponding to a block is time-sequentially provided.

  According to this aspect, it is possible to create sequence data for rap performance by a simple operation of laying out blocks indicating phrases in the sequence data editing area.

It is a block diagram which shows the structure of the song synthesizing | combining apparatus which is 1st Embodiment of this invention. It is a block diagram which shows the structure of the phrase data generation part 100 in the same embodiment. It is a block diagram which shows the structure of the song synthesis | combination part 200 in the embodiment. It is a figure explaining operation | movement of the embodiment. It is a block diagram which shows the structure of the song synthesizing | combining apparatus which is 2nd Embodiment of this invention. It is a time chart which shows operation | movement of the embodiment. It is a block diagram which shows the structure of the song synthesizing | combining apparatus which is 3rd Embodiment of this invention. It is a figure which shows the edit process of the sequence data in the embodiment. It is a figure which shows the piano roll display used in other embodiment of this invention. It is a figure explaining the volume operation of the phrase in other embodiment of this invention. It is a figure explaining the block layout performed in other embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram illustrating a configuration example of a singing voice synthesis apparatus according to the first embodiment of the present invention. This singing synthesizer is obtained by installing a singing synthesizer program on a mobile terminal having a function of outputting sound, such as a portable game machine, a mobile phone, or a smartphone. As shown in FIG. 1, the song synthesizer includes a CPU 10, a nonvolatile memory 20, a ROM 30, a RAM 40, a display input unit 50, a sound system 60, and an external interface group 70.

  CPU10 is a control center which controls each part of this song synthesis apparatus. The ROM 30 is a read-only memory that stores a control program for controlling basic operations of the singing voice synthesizing apparatus such as a loader. The RAM 40 is a volatile memory used as a work area by the CPU 10.

  The display input unit 50 is, for example, a touch panel, and is a user interface having a display function for displaying an operation state of the apparatus, input data, a message for an operator (user), and the like, and an input function for receiving an operation performed by the user. . The contents of the operation performed by the user include input of information indicating lyrics, input of information indicating notes, input of an instruction to reproduce a synthesized singing sound for one phrase, and the like. The display input unit 50 has a configuration in which the display function and the input function are integrated, such as a touch panel, and a configuration in which the display function and the input function are separated, such as a display and a keyboard. May be.

  The external interface group 70 includes a network interface for performing data communication with other devices via a network, a driver for transferring data with an external storage medium such as a flash memory, and the like.

  The sound system 60 is means for outputting, as speech, time-series digital data indicating the waveform of the synthesized singing sound obtained by the singing synthesizer, and converts the time-series digital data indicating the waveform of the synthesized singing sound into an analog voice signal. A D / A converter, an amplifier that amplifies the analog audio signal, a speaker that outputs the output signal of the amplifier as sound, and the like are included.

  The nonvolatile memory 20 is a storage device for storing information such as various programs and databases. For example, an EEPROM (Electrically Erasable Programmable Read Only Memory; ROM that can be electrically erased and written) is used. Among the contents stored in the nonvolatile memory 20, there are a singing synthesis program 21, a dictionary 25, a phrase database 26, and a speech segment database 28 as specific to the present embodiment. The CPU 10 loads the program in the non-volatile memory 20 into the RAM 40 and executes it in accordance with an instruction input via the display input unit 50.

  The program stored in the nonvolatile memory 20 may be traded by downloading via a network. In this case, the program or the like is downloaded from a site in the Internet via an appropriate one in the external interface group 70 and installed in the nonvolatile memory 20. The program or the like may be traded in a state stored in a computer-readable storage medium. In this case, the program or the like is installed in the nonvolatile memory 20 via an external storage medium such as a flash memory.

  The feature of this embodiment is in the song synthesis program 21. The song synthesis program 21 includes a phrase data generation unit 100 and a song synthesis unit 200. The phrase data generation unit 100 receives the lyric data via the display input unit 50 that is a user interface, divides the lyrics indicated by the lyric data into a plurality of fragments, generates a phonetic symbol string from each fragment, and generates each phonetic symbol string. Is a program for generating a plurality of phrase data corresponding to a plurality of phrases, respectively. Here, the phrase is a fragmentary singing voice that is quickly pronounced when singing by rap, and the phrase data is time-series data indicating the utterance mode of the phrase. In the present embodiment, the dictionary 25 is referred to when generating this phrase data. Each phrase data generated by the phrase data generation unit 100 is stored in the phrase database 26. The singing voice synthesizing unit 200 is a means for synthesizing the singing sound from the phrase data.

  FIG. 2 is a block diagram illustrating a configuration of the phrase data generation unit 100. In FIG. 2, in order to facilitate understanding of functions of each unit of the phrase data generation unit 100, a dictionary 25, a phrase database 26, and the like related to these units are also illustrated.

  The phrase data generation unit 100 in the present embodiment includes a lyrics analysis unit 101, a phonetic symbol generation unit 102, and a phrase data synthesis unit 103 shown in FIG. The lyric analysis unit 101 stores the lyric data input by the operation of the display input unit 50 in the work area in the RAM 40, and divides the lyrics indicated by the lyric data into fragments, specifically, phrases. Here, the phrase is a unit of pronunciation in which an adjunct (noun, verb, etc.) is connected to an adjunct (a particle etc.) (there may be no adjunct).

  The lyric analysis unit 101 refers to the dictionary 25 when dividing the lyrics into phrases. The dictionary 25 is a collection of word data and rule base. The word data includes information indicating the part of speech of a word such as a noun, a verb, or a particle, and pronunciation symbol information corresponding to the word. Furthermore, the word data includes a pitch model that indicates a normal pitch change when the word is pronounced. In addition to the information defined for each word, the dictionary 25 includes rules for obtaining the pitch trajectory of the uttered sound when a person normally utters the sentence based on the structure of various sentences. .

  Then, the lyric analysis unit 101 compares the word data in the dictionary 25 with the character string constituting the lyric to obtain the top position of each word included in the lyric. Next, the lyrics analyzing unit 101 determines the part of speech of each word included in the lyrics from the part of speech information included in the word data of the dictionary 25, and distinguishes the word into an independent word and a close word. Then, the division position of the phrase in the lyrics is determined from the order of distinct independent words and adjuncts, and the lyrics are divided into phrases. Furthermore, the lyric analysis unit 101 refers to the dictionary 25 to obtain a pitch trajectory of the uttered sound when a person normally utters the lyrics, and classifies the pitch trajectory for each phrase.

  The plurality of phrases generated from the lyrics by the lyrics analyzing unit 101 are delivered to the phonetic symbol generating unit 102. In this embodiment, this phrase becomes the material of the phrase sung during the lap. The phonetic symbol generation unit 102 refers to the dictionary 25 to convert each phrase delivered from the lyrics analysis unit 101 into a phonetic symbol string.

  The phrase data synthesizing unit 103 synthesizes phrase data for each phrase obtained by dividing the lyrics using the phonetic symbol string obtained by the phonetic symbol generating unit 102. This phrase data is composed of a phonetic symbol track and a pitch track having a common time axis. The phonetic symbol track is a track for designating the pronunciation timing of each phonetic symbol in the phonetic symbol string. The phrase data synthesis unit 103 maps each phonetic symbol constituting the phonetic symbol string to the phonetic symbol track, for example, with a certain time interval. The pitch track is a track showing a change in pitch over time during a phrase. The pitch track generation method differs depending on what kind of lap is to be performed by the synthesizer.

  The song synthesizer according to this embodiment can perform three types of laps. The first lap, which is a normal lap that does not change the pitch very much, the second lap that utters like a singing by changing the pitch, and the third lap that utters like a song. The user can specify which type of lap is to be performed by operating the display input unit 50.

  When the user instructs the first lap, the phrase data synthesis unit 103 maps a constant pitch from the start point to the end point of the phrase on the pitch track. In a preferred embodiment, in order to make the lap natural, the pitch mapped to the pitch track of each phrase data is changed using a pseudo-random number or the like to give subtle variations in pitch between phrases.

  When the user instructs the second lap, the phrase data synthesizing unit 103 refers to the phrase that is the source of the phrase data and the dictionary 25, and indicates a pitch model that indicates a pitch change when a word included in the phrase is pronounced. Is read from the dictionary 25 and this pitch model is mapped to the pitch track.

  When the user instructs the third lap, the phrase data synthesis unit 103 maps the pitch trajectory of the phrase that is the basis of the phrase data among the pitch trajectories of each phrase generated by the lyrics analysis unit 101 to the pitch track.

  As described above, in this embodiment, phrase data is generated for each phrase in which the lyrics indicated by the lyrics data are segmented. Each phrase and the phrase data obtained from those phrases are stored in the phrase database 26 together with the original lyric data.

  As shown in FIG. 2, the area for storing the phrase database 26 is divided into a lyrics data area and a phrase data area. In the lyrics data area, the lyrics data to be processed by the phrase data generation unit 100 is stored together with an ID for identifying the lyrics data. In the phrase data area, the ID of the lyric data to which the phrase belongs, the serial number FN in the lyrics of the phrase, the phrase, and the phrase are obtained for each phrase into which the lyrics indicated by the lyrics data are divided. Stored phrase data.

  In FIG. 1, a singing voice synthesizing unit 200 receives a reproduction instruction for a desired phrase data via the display input unit 50 as a user interface, reads out the phrase data that is the object of the reproduction instruction from the phrase database 26, and This is a program for synthesizing time-series digital data indicating a waveform of a synthesized singing sound of a phrase using phrase data. In the present embodiment, the speech segment database 28 is referred to when generating time-series digital data of this phrase.

  The speech segment database 28 is a collection of speech segment data indicating various speech segments that are singing voice materials. These speech segment data are data created based on speech segments extracted from speech waveforms emitted by actual humans. In the speech segment database 28, a group of speech segment data obtained from the singer speech waveform of each singer is prepared for each singer with different voice quality. At the time of song synthesis by the song synthesis unit 200, the user can select a group of speech segment data to be used for song synthesis from a group of various speech segment data.

  FIG. 3 is a block diagram illustrating a configuration of the singing voice synthesizing unit 200. As shown in FIG. 3, the singing voice synthesizing unit 200 includes software modules such as a reading control unit 220, a pitch conversion unit 230, a segment connection unit 240, and a volume control unit 250.

  The read control unit 220 receives a reproduction instruction for a desired phrase data via the GUI, a function for providing a user with a GUI for receiving a reproduction instruction for the phrase data via the display input unit 50, And a function of performing control for reproducing the phrase data that is the target of the reproduction instruction as a phrase.

  First, the former function will be described. The reading control unit 220 performs a process for specifying the lyric data that the user desires to sing by rap. Various types of processing modes can be considered. For example, a character string corresponding to, for example, a predetermined number of characters at the beginning of the lyrics indicated by each lyric data for which phrase data has been generated is displayed on the display input unit 50 as a menu. The user may select desired lyric data. Alternatively, a part of desired lyrics may be input as a keyword, and lyrics data including the keyword may be displayed on the display input unit 50 as a menu, and desired lyrics data may be selected from the menu.

  The reading control unit 220 reads the ID of the lyrics data selected by the user in this way from the lyrics data area of the phrase database 26, and each phrase / phrase data pair associated with the same ID as this ID is stored in the phrase database. 26 phrase data areas are read out and stored in the work area in the RAM 40. And the block which shows visually the content of each clause in a work area is displayed on the display input part 50. FIG. Thus, the user can input a reproduction instruction for phrase data of a desired phrase by instructing a block of the desired phrase displayed on the display input unit 50.

Next, the latter function will be described. When the reading control unit 220 detects that one of the blocks displayed on the display input unit 50 is designated, the reading control unit 220 identifies the phrase data corresponding to the designated block, and this phrase data in the work area. The phonetic symbol and pitch data are read out from the phonetic symbol track and the pitch track, respectively. Further, the reading control unit 220 reads out the speech unit data corresponding to the read phonetic symbol from the speech unit database 28. Then, the read control unit 220 sends the read speech element data and pitch data to the pitch conversion unit 230.
The above is the function of the read control unit 220.

  The pitch conversion unit 230 is a unit that performs pitch conversion on the speech segment data read by the read control unit 220. The pitch conversion unit 230 performs pitch conversion of the speech unit data according to the pitch data read by the read control unit 220 together with the speech unit data, and generates speech unit data having a pitch indicated by the pitch data.

  The unit linking unit 240 is a unit that performs a process of smoothly connecting the speech unit data that has undergone the processing of the pitch conversion unit 230 and outputs it as time-series data indicating a singing speech waveform.

The volume control unit 250 controls the volume of the time-series data output from the segment connecting unit 240 in accordance with a volume setting operation performed using the display input unit 50, for example, and shows a time series indicating the final synthesized singing sound It is a means for outputting to the sound system 60 as data.
The above is the configuration of the singing voice synthesizing apparatus according to the present embodiment.

  FIG. 4 is a diagram illustrating an operation example of the present embodiment. The operation of this embodiment will be described below with reference to FIG. When the user wants the song synthesizer to perform rap with desired lyrics, first, the user inputs lyrics data indicating the lyrics by operating the display input unit 50. FIG. 4A shows the lyrics “Eiji ran. And Eiji flew with full power. Ah.” Indicated by the lyric data input by this operation. This lyrics data is stored in the work area in the RAM 40.

  When the input of the lyrics data is completed, the user inputs a phrase data generation instruction by designating a desired one of the first to third laps described above by operating the display input unit 50. As a result, the lyric analysis unit 101 of the phrase data generation unit 100 refers to the dictionary 25 and divides the lyric data in the work area into phrases as shown in FIG. Ask. In FIG. 4B, the symbol “|” indicates a segment break.

  Each phrase divided in this way is delivered to the phonetic symbol generation unit 102. In the phonetic symbol generation unit 102, each clause delivered from the lyrics analysis unit 101 is converted into a phonetic symbol string.

  Then, the phrase data synthesis unit 103 generates phrase data for each phrase. At that time, as described above, pitch mapping is performed on the pitch track of the phrase data by a different method depending on the type of lap instructed by the user. FIG. 4C illustrates the contents of the pitch track of the phrase data generated for each phrase in this way. Each phrase and phrase data corresponding to each phrase are stored in the frace database 26 together with the lyrics data.

  When making the song synthesizer execute a song by rap, the user uses the display input unit 50 to perform an operation for selecting lyrics data. When the user selects the lyric data shown in FIG. 4 (A), the reading control unit 220 of the singing composition unit 200 causes each phrase of the lyrics indicated by the lyric data shown in FIG. 4 (A) and the phrase corresponding to each phrase. Data is read from the phrase database 26 and stored in the work area in the RAM 40. Then, a block that visually indicates each phrase stored in the work area is displayed on the display input unit 50. In this example, as illustrated in FIG. 4D, a block B1 indicating the phrase “Eiji”, a block B2 indicating the phrase “run”, a block B3 indicating the phrase “and”, and a phrase “Eiji is”. A block B4 showing, a block B5 showing the phrase “with full power”, a block B6 showing the phrase “flew”, and a block B7 showing the phrase “oh” are displayed on the display input unit 50. In this way, the preparation for responding to the reproduction instruction from the user is completed.

  In this state, the user can cause the singing apparatus to perform singing by rap by tapping a block indicating a desired phrase (sentence) displayed on the display input unit 50. Here, tapping is an operation of tapping the displayed block with a fingertip.

  FIG. 4E shows an execution example of phrase singing synthesis executed in response to this tap. In this example, the blocks B1, B1, B5, B7, B7, and B6 are tapped at times T10, T11, T12, T13, T14, and T15.

  At time T10, when the block B1 indicating the phrase “Eiji wa” is tapped, the reading control unit 220 identifies the phrase data of the phrase “Eiji wa” corresponding to the tapped block B1, and within the work area. The phonetic symbol and pitch data are read out from the phonetic symbol track and pitch track of the phrase data, respectively. Further, the reading control unit 220 reads out the speech unit data corresponding to the read phonetic symbol from the speech unit database 28. Then, the read control unit 220 sends the read speech element data and pitch data to the pitch conversion unit 230.

  The pitch conversion unit 230 performs pitch conversion on the speech unit data read by the read control unit 220, and generates speech unit data having a pitch indicated by the pitch data read by the read control unit 220. The segment linking unit 240 performs a process of smoothly connecting the speech segment data that has undergone the processing of the pitch conversion unit 230, and outputs time-series data indicating a singing speech waveform. The volume control unit 250 controls the volume of the time series data output from the segment connection unit 240 and outputs the time series data indicating the final synthesized singing sound. In this way, the synthesized singing sound of the phrase “Eijiha” is emitted.

  The same applies to the following. When the block B1 is tapped at the time T11, the synthesized singing sound of the phrase “Eijiha” is emitted, and when the block B5 is tapped at the time T12, the synthesized singing sound of the phrase “full power”. Is emitted, and when the block B6 is tapped at time T15, the synthesized singing sound of the phrase “flew” is emitted.

  If a block is tapped and the next tap is performed while the phrase data corresponding to the tapped block is being emitted as a synthesized singing sound, the pronunciation of the previous phrase data is You may make it generate | occur | produce the synthetic | combination singing sound based on the phrase data instruct | indicated by the next tap which is complete | finished at the time. For example, when the block B1 indicating “Eiji wa” is tapped and sounded up to “English”, when the block B1 indicating “Eiji wa” is tapped again, the sound is generated as “Eiji wa wa”. In this way, it is possible to obtain an effect similar to scratch (a technique for repeatedly reproducing the same portion used in a disc jockey), and it is possible to easily rap a song with a more rhythmic feeling.

  As described above, according to the present embodiment, when the user inputs lyrics data of desired lyrics to the song synthesizer, phrase data is generated from each phrase of the lyrics indicated by the lyrics data by the phrase data generation unit 100 of the song synthesizer. Is generated. Then, the singing voice synthesizing apparatus causes the display input unit 50 to display blocks indicating each phrase of the lyrics for the user who desires the rap performance. In this state, by tapping a block indicating a desired phrase at a desired timing, the user can emit the phrase data of the phrase corresponding to the block as a synthesized singing sound. Therefore, the user can lap using the singing voice synthesizing apparatus according to the present embodiment.

Second Embodiment
Next, a singing voice synthesizing apparatus according to a second embodiment of the present invention will be described. The singing voice synthesizing apparatus in the present embodiment is obtained by replacing the singing voice synthesizing unit 200 of the singing voice synthesizing program in the first embodiment with a singing voice synthesizing unit 200A shown in FIG. This singing composition unit 200A has a configuration in which a pitch change control unit 320 is added to the singing composition unit 200 in the first embodiment.

  The singing voice synthesizing unit 200 according to the first embodiment outputs phrase data corresponding to the block as a synthesized singing sound in response to a playback instruction to tap the block displayed on the display input unit 50. On the other hand, according to the reproduction | regeneration instruction | indication which flicks the block displayed on the display input part 50, 200 A of song synthesis | combination parts by this embodiment output the phrase data corresponding to the block as a synthetic song sound. Here, the flick is an operation of lightly removing the block displayed on the display input unit 50. When the song synthesis unit 200A in the present embodiment generates a synthesized song sound using phrase data corresponding to a flicked block, the song synthesis unit 200A gives a pitch change in a direction corresponding to the direction of the flick to the synthesized song sound. The pitch change control unit 320 provided in the singing voice synthesis unit 200A is a means for giving a pitch change corresponding to the direction of the flick to the synthesized singing sound.

  In the present embodiment, the reading control unit 220 and the pitch change control unit 320 are supplied with information indicating the contact position of the finger on the screen of the display input unit 50. When a certain block is flicked, the read control unit 220 identifies the flicked block based on the first contact position information generated by the flick, and uses the phrase data corresponding to the block as a reproduction target. As described above, in this embodiment, when a finger first touches a certain block, the phrase data corresponding to that block is determined as a reproduction instruction target. After this determination, unless the finger is separated from the screen of the display input unit 50, the reproduction instruction target does not change regardless of how the finger contact position moves. Therefore, the user can perform flicking without being restricted by the block frame.

  When the phrase data that is the target of the reproduction instruction is determined in this way, the reading control unit 220 reads the phonetic symbol and pitch data from the phonetic symbol track and the pitch track of the phrase data, and speech segment data corresponding to the phonetic symbol. Are read from the speech segment database 28. Then, the read control unit 220 sends the speech segment data to the pitch conversion unit 230 and sends the pitch data to the pitch change control unit 320.

  The pitch change control unit 320 obtains the direction of the flick based on the change in the contact position indicated by the contact position information, and generates a pitch change amount that changes with a time gradient corresponding to the direction of the flick. Specifically, when the direction of the flick is diagonally right upward, the pitch change control unit 320 generates a pitch change amount that changes with a positive time gradient. When the direction of the flick is diagonally downward to the right, the pitch change control unit 320 generates a pitch change amount that changes with a negative time gradient. When the direction of the flick is the horizontal direction, the pitch change control unit 320 generates 0 as the pitch change amount. When flicking in the upper right direction or the lower right direction, the absolute value of the time gradient of the pitch change amount may be increased as the angle formed by the flick direction with respect to the horizontal direction is increased. Then, the pitch change control unit 320 outputs pitch data obtained by adding the pitch change amount to the pitch data given from the read control unit 220 to the pitch conversion unit 230. The functions of the pitch conversion unit 230, the segment connection unit 240, and the volume control unit 250 are as described in the first embodiment.

  FIG. 6 is a time chart showing an operation example of the present embodiment. In this example, the blocks B1, B3, B1, and B7 are flicked at times T20, T21, T22, and T23, respectively. At times T20 and T22, the block B1 is flicked in the horizontal direction. Therefore, at times T20 and T22, a synthesized singing sound having the pitch specified by the pitch track of the phrase data is generated as the synthesized singing sound of the phrase “Eijiha” corresponding to the block B1. At time T21, the block B3 is flicked diagonally to the lower right. Therefore, at this time T21, a pitch change amount having a negative time gradient is generated, and a synthesized singing sound whose pitch decreases toward the end of the phrase “and” corresponding to the block B3 is generated. The At time T23, the block B7 is flicked obliquely in the upper right direction. Therefore, at time T23, a pitch change amount having a positive time gradient is generated, and a synthetic singing sound whose pitch increases toward the end of the phrase “Ah” corresponding to block B7 is generated. The

  As described above, according to the present embodiment, when a block displayed on the screen is flicked, the phrase corresponding to the flicked block is changed in pitch according to the direction of the flick on the basis of the pitch of the phrase data. Pronounced. For this reason, the pitch of the ending of the phrase can be increased or decreased, and the user can perform live performance with inflection on the pronunciation of the phrase by a simple operation of flicking.

  When changing the pitch of a phrase by flicking, the pitch may be changed from the beginning to the end of the phrase, but the pitch is changed from the starting point of the pitch change in the middle of the phrase to the end. You may make it let it. In the latter case, the part from the beginning of the phrase to the pitch change start point is sounded at the pitch specified by the pitch track of the phrase data, and the part from the pitch change start point to the end of the phrase changes according to the flick operation. Is given and pronounced. In this case, the user may be able to specify the pitch change start point in the phrase by operating the display input unit 50.

  Alternatively, the speed of flicking (fast or slow) may be detected and used as a phrase pronunciation parameter. For example, the speed of flick is made to correspond to the speed of reproducing a phrase. Further, the flick distance may be detected and used as a sound generation parameter. For example, when a flick over a long distance is detected, the phrase phrase is played back to the end, and when a short distance flick is detected, the phrase phrase is played back halfway.

  In addition to simple flick operations in the horizontal direction and diagonally right and up and down, various flick operations may be recognized and reflected in pitch changes. For example, the pitch may be raised from the middle of the phrase by moving the contact position of the finger in the right lateral direction to the middle of the flick and moving the finger diagonally upward to the right from the middle. Alternatively, the phrase may be given a pitch change that repeatedly rises and lowers in small increments, such as vibrato, by moving the contact position of the finger in a wave shape. In this case, the pronunciation of the phrase may be maintained for the duration of the flick (the time during which the finger continues to touch the screen of the display input unit 50). For example, if the block corresponding to the phrase “Eijiha” is flicked and the duration of the flick becomes longer, the vowel “a” that is the vowel of “Eijiha” is extended longer and the pronunciation duration of the phrase In the meantime, a complicated pitch change may be given to the voice of “A” in accordance with a complicated flick operation.

<Third Embodiment>
FIG. 7 is a block diagram showing a configuration of a singing voice synthesizing apparatus according to the third embodiment of the present invention. In the singing voice synthesizing apparatus according to the present embodiment, the non-volatile memory 20 stores a singing voice synthesis program 21B and a sequence database 27. The song synthesis program 21B includes the phrase data generation unit 100 in the first and second embodiments, the song synthesis unit 200A in the second embodiment, the sequence data editing unit 300, and the sequence data reproduction unit 400.

  The sequence data editing unit 300 is a program that edits the sequence data in accordance with the operation of the display input unit 50 and stores it in the sequence database 27. This sequence data is time-series data in which phrase data playback instructions are mapped on the time axis. In the present embodiment, the reproduction instruction for one phrase data includes information specifying the phrase data (for example, the combination of the lyrics ID and the serial number FN in the first embodiment) and the pitch change described in the second embodiment. Contains information that indicates the generation of the quantity. The sequence data reproducing unit 400 reads out the sequence data designated by the user from the sequence database 27, reproduces the sequence data, and synthesizes the phrase data that is the target of the reproduction instruction in accordance with the reproduction instruction of the phrase data in the sequence data. It is a program that outputs as sound.

  Here, the function of the sequence data editing unit 300 in this embodiment will be described with reference to FIG. The sequence data reproducing unit 4 according to the present embodiment reproduces sequence data using phrase data in the phrase database 26 as will be described later. Therefore, in order for the sequence data to be reproducible, phrase data for which reproduction is instructed in the sequence data must be stored in the phrase database 26. On the other hand, the lyrics data area of the phrase database 26 stores the lyrics data that is the basis of the phrase data in the phrase data area (see FIG. 2). Therefore, the sequence data editing unit 300 in the present embodiment causes the user to perform an operation for selecting lyric data including a desired phrase (sentence). When the user selects, for example, the lyric data shown in FIG. 4A, the sequence data editing unit 300 reads a phrase and phrase data set related to the lyric data from the phrase data area of the phrase database 26, and FIG. , Block B1 indicating the phrase “Eiji is”, block B2 indicating the phrase “run”, block B3 indicating the phrase “and”, block B4 indicating the phrase “Eiji is”, and the phrase “with full power” Are displayed on the display input unit 50. The block B5 indicating the phrase “Flew” and the block B7 indicating the phrase “Oh” are displayed on the display input unit 50. The sequence data editing unit 300 displays a sequence data editing area below the display area of these blocks. This sequence data editing area is an area in which the horizontal axis represents time and the vertical axis represents pitch.

  In a state where this sequence data editing area is displayed, the user can edit the sequence data. Specifically, when the user touches a block corresponding to a desired phrase and moves the contact position, a copy of the block moves following the contact position. Then, the user moves the copy of the block to a desired position on the time axis of the sequence data editing area, thereby instructing the sequence data editing unit 300 to generate the phrase data playback instruction corresponding to the block. can do.

  In this state, when the user flicks the right end portion of the copy of the block on the time axis from the top to the bottom, the copy of the block is inclined downward to the right. In this way, the user can add a pitch change amount generation instruction that changes with a negative gradient to the phrase data reproduction instruction indicated by the copy of the block. Further, when the user flicks the left end portion of the copy of the block on the time axis from the top to the bottom, the copy of the block is inclined upward. In this way, the user can add a pitch change amount generation instruction that changes with a positive gradient to the phrase data reproduction instruction indicated by the copy of the block.

  In the example shown in FIG. 8, on the time axis, in order from the left, block CB1a that is a copy of block B1, block CB1b that is a copy of block B1, block CB5 that is a copy of block B5, block CB7a that is a copy of block B7, A block CB7b that is a copy of the block B7 and a block CB6 that is a copy of the block B6 are arranged. Here, after the blocks CB1a, CB5, and CB7b are arranged on the time axis, the flick operation is not performed and the horizontal posture is maintained. Accordingly, an instruction to generate a pitch change amount = 0 is added to the reproduction instruction of phrase data corresponding to these. The blocks CB1b and CB6 are inclined downward by a flick operation after being arranged on the time axis. Therefore, an instruction to generate a pitch change amount that changes with a negative gradient is added to the reproduction instruction of phrase data corresponding to these. The block CB7a is inclined upward by a flick operation after being arranged on the time axis. Therefore, an instruction to generate a pitch change amount that changes with a positive gradient is added to the corresponding phrase data reproduction instruction.

  The sequence data editing unit 300 generates sequence data based on the blocks laid out in the sequence data editing area in this way, and stores them in the sequence database 27. That is, the generation timing of the phrase data playback instruction corresponding to each block is determined based on the position of each block in the time axis direction, and the amount of pitch conversion to be added to the phrase data of each block is generated based on the direction of each block The contents of the instruction are determined, and the phrase data reproduction instruction obtained in this way is time-series to generate sequence data.

  The sequence data reproducing unit 400 reads out the sequence data designated by the operation of the display input unit 50 from the sequence database 27, scans the sequence data from the top, takes out the reproduction instruction of the phrase data, and extracts the phrase data reproducing unit 200A (FIG. 5). To the read control unit 220). Also, a pitch change amount generation instruction included in the reproduction instruction is given to the pitch change control unit 320.

  In the second embodiment, the pitch change control unit 320 generates the pitch change amount based on the flick direction. On the other hand, when the sequence data is reproduced in the present embodiment, the pitch change control unit 320 generates the pitch change amount according to the pitch change amount generation instruction extracted from the phrase data reproduction instruction instead of the flick direction. The pitch change control unit 320 sends the pitch data obtained by adding the pitch change amount to the pitch data given from the read control unit 220 to the pitch conversion unit 230. The operations of the pitch conversion unit 230, the segment connection unit 240, and the volume control unit 250 are as described above.

  According to the present embodiment, as described with reference to FIG. 8, a desired phrase is reproduced with a desired pitch change at a desired timing by performing a simple block movement operation and rotation operation. Can be generated, and lap reproduction can be performed using this sequence data.

<Other embodiments>
Although the first to third embodiments of the present invention have been described above, other embodiments are conceivable for the present invention. For example:

  (1) In each of the above embodiments, the lyrics are divided into phrases, and phrases are generated from the phrases. However, one phrase may be generated from a combination of a plurality of clauses by the user setting or editing. Moreover, in each said embodiment, although the lyrics were divided | segmented into the phrase, the way of dividing a lyrics is not limited to this. For example, in the case of English lyrics, the lyrics may be divided into words, and a phrase may be generated from each word. That is, the lyrics may be divided into fragments according to an arbitrary rule, and a phrase may be generated from each fragment.

  (2) In the third embodiment, as shown in FIG. 9, a piano roll display composed of a piano keyboard image 301 and a sequence data editing area 302 on the right side is adopted as means for receiving sequence data input. May be. Here, in the sequence data editing area 302, the horizontal axis represents time, and the vertical axis represents pitch (each key of the piano keyboard).

  In the example shown in FIG. 9, a block CB21 indicating the phrase “Eijiha”, a block CB22 indicating the phrase “with full power”, and a block CB23 indicating the phrase “flyed” are arranged in the sequence data editing area 302. In this aspect, the sequence data editing unit generates sequence data in which phrase data reproduction instructions including a pitch offset generation instruction are time-series in addition to the pitch change generation instruction.

  Here, in the phrase data reproduction instruction corresponding to each block, the pitch offset instructed by the pitch offset generation instruction is the vertical position on the staff score of each block (or the piano key located on the left side of each block). ) And the time gradient of the pitch change amount indicated by the pitch change amount generation instruction is determined based on the inclination of each block.

  When the sequence data is reproduced, the pitch change control unit 320 (see FIG. 5) of the singing synthesis unit 200A generates a pitch offset according to the pitch offset generation instruction included in the reproduction instruction extracted from the sequence data, and the pitch change amount A pitch change amount is generated in accordance with the generation instruction. Then, the pitch offset of the phrase data to be reproduced is ignored, the pitch offset and the pitch change amount are added, and the pitch data as the addition result is output to the pitch conversion unit 230.

  According to this aspect, when reproducing the sequence data, when the phrase data reproduction instruction generated based on the block CB21 is read from the sequence data, the phrase “Eijiha” is pronounced at the pitch of the black key KY1. When the phrase data playback instruction generated based on the block CB22 is read from the sequence data, the phrase “full power” is pronounced at a pitch that starts from the pitch of the black key KY2 and decreases toward the end of the phrase. The When the phrase data reproduction instruction generated based on the block CB23 is read from the sequence data, the phrase “flew” is pronounced at the pitch of the white key KY3. According to this aspect, the pitch of the phrase can be changed greatly in sequence reproduction, and the width of the rap performance can be widened.

  (4) In the second embodiment, the speed at which the block is flicked may be detected, and the sound volume when the phrase is pronounced may be controlled according to the speed.

  (5) In the third embodiment, when editing sequence data, the shape of the block to be laid out in the sequence data editing area is changed according to the operation performed by the user, and the block is adapted to the shape of this block. You may make it change the sound volume at the time of pronounce | playing the phrase which carried out.

  FIG. 10 is a diagram for explaining volume control by deformation of this block. In this example, the sequence data editing unit 300 causes the display input unit 50 to display two kinds of volume change instruction buttons SB1 and SB2.

  Here, it is assumed that the user designates the volume change instruction button SB1 after instructing the block CB30 indicating, for example, the phrase “Eijiha” in the sequence data editing area. In this case, the sequence data editing unit 300 transforms the block CB30 into a block CB30a that becomes thicker toward the right, generates volume control data that increases the volume toward the end of the phrase, and adds it to the phrase data playback instruction. To do.

  On the other hand, it is assumed that the user designates the volume change instruction button SB2 after instructing the block CB30 in the sequence data editing area. In this case, the sequence data editing unit 300 transforms the block CB30 into a block CB30b that becomes narrower toward the right, and generates volume control data that reduces the volume toward the end of the phrase and adds it to the phrase data playback instruction. To do.

  When the sequence data is reproduced, the volume of the phrase is controlled according to the volume control data added to the phrase data reproduction instruction in this way. According to this aspect, in addition to the pitch change, a volume change can be given to the phrase, and a powerful rap performance can be performed.

  (6) The aspect which combined the said 1st Embodiment and 2nd Embodiment can be considered. In this aspect, when the block displayed on the display input unit 50 is tapped, the singing composition unit generates a phrase using phrase data corresponding to the tapped block, as in the first embodiment. Further, when the block displayed on the display input unit 50 is flicked, the singing composition unit generates a phrase using phrase data corresponding to the flicked block as in the second embodiment, and at that time The pitch change according to the flick direction is given to the phrase. In this aspect, when the block is tapped, the singing composition unit generates a phrase having a length of time determined by the phrase data corresponding to the block, and when the block is flicked, the phrase data corresponding to the block is generated. May be used to generate a phrase that lasts for a time corresponding to the duration of the flick.

  (7) In the first and second embodiments, whether or not the timing at which the user taps or flicks a block is appropriate (for example, whether or not the user taps or flicks in synchronization with a certain rhythm) is evaluated. You may provide the function which displays a result in a song synthesizing | combining apparatus. Alternatively, when a tap or flick is performed at an inappropriate timing, the singing synthesis apparatus may be provided with a correction function that treats the tap or flick as being performed at an appropriate timing immediately after that.

  (8) The aspect which introduce | transduced the technique of the said 3rd Embodiment to the said 1st Embodiment can be considered. In this aspect, an operation of tilting a desired block can be performed in a state where the block is displayed on the display input unit 50 as shown in FIG. 4D. When the block is tilted, the singing composition unit generates a time-series pitch change amount that changes with a time gradient corresponding to the block inclination, and the time-series pitch change amount corresponds to the block. The phrase data is added to the pitch data of the pitch track.

  When this block is tapped, a phrase having a pitch change corresponding to the inclination of this block is generated. By performing the operation of tilting the block in this way, a phrase having a desired pitch change can be generated. In this aspect, a copy of a block may be generated, and an operation for tilting the block that is the copy may be performed. When a copy of a block tilted in this way is generated, a copy of the phrase data corresponding to the original block is generated, and a pitch change amount corresponding to the block inclination is added to the pitch track of the phrase data that is the copy. is there. When the original block is tapped, the phrase is synthesized using the phrase data corresponding to the original block, and when the tilted copy block is tapped, the phrase is copied using the phrase data of the copy. Is synthesized. According to this aspect, phrases of the same phrase can be synthesized with various pitch changes, and various performances can be performed.

  (9) As another aspect in which the technique of the third embodiment is introduced into the first embodiment, the following is conceivable. First, as shown in FIG. 11, a piano keyboard image 301 and a block layout area 303 on the right side thereof are displayed on the display input unit 50. Here, in the block layout area 303, the vertical axis is the pitch (or each key of the piano keyboard), but there is no horizontal axis. The block layout area 303 has a horizontal extent so that a plurality of blocks can be arranged side by side. In this aspect, the user can place a block corresponding to a desired phrase in the block layout area 303. At that time, the block can be tilted. In the example shown in FIG. 11, the blocks CB21 to CB26 are arranged in the block layout area 303.

  In this state, it is assumed that the user taps the block CB25 corresponding to the phrase “with full power”. In this case, the song synthesis unit generates a pitch offset corresponding to the white key KY3 on the left side of the block CB25. Further, since the block CB25 faces the horizontal direction, the singing composition unit generates 0 as the pitch change amount. Then, when synthesizing the singing sound of the phrase corresponding to the block CB25, the singing synthesis unit ignores the pitch track of the phrase data corresponding to the block CB25, and the pitch offset and the pitch change amount corresponding to the white key KY3 = 0. Is added and a singing sound having a pitch corresponding to the addition result is synthesized. Accordingly, in this case, the phrase “with full power” is pronounced at the pitch of the white key KY3.

  Further, it is assumed that the user taps a block CB22 corresponding to the phrase “full power” and tilted downward. In this case, the song synthesis unit generates a pitch offset corresponding to the black key KY2 on the left side of the block CB22. Further, since the block CB22 is inclined downward, the singing composition unit generates a pitch change amount that changes with a negative time gradient. Then, when the singing sound of the phrase corresponding to the block CB22 is synthesized, the singing synthesis unit adds the pitch offset corresponding to the black key KY2 and the pitch change amount that changes with a negative time gradient, and corresponds to this addition result. Synthesize a singing sound with a pitch to play. Therefore, in this case, the phrase “full power” is pronounced starting from the pitch of the black key KY2 and decreasing toward the end of the word.

  (10) In each of the above embodiments, when a phrase reproduction instruction is given, the singing synthesizing unit synthesizes time-series data indicating the singing voice waveform from the phrase data. However, instead of performing singing synthesis during playback in this way, when the lyrics are divided into fragments, time-series data indicating the singing voice waveform of the fragment is generated as the phrase data of each fragment, and this is used as the phrase data. When it is stored and a reproduction instruction is generated, the target phrase data may be output as it is from the sound system.

  (11) In the above embodiments, when a block is rotated 180 degrees by flicking, the phrase corresponding to the block may be reproduced in reverse.

  (12) In the third embodiment, the singing voice synthesizing apparatus having the phrase data generation function, the phrase data reproduction function in response to the tap or flick, the sequence data editing function, and the sequence data reproduction function has been described. However, a sequence data editing apparatus having only a sequence data editing function may be configured.

  (13) The song synthesizer in each of the above embodiments stores the dictionary 25, the phrase database 26, the speech segment database 28, and the sequence database 27 in the nonvolatile memory 20. However, the dictionary 25, the phrase database 26, the speech segment database 28, and the sequence database 27 are stored in an external storage means, not the storage means provided in such a singing voice synthesizing apparatus, and these are stored via a network such as the Internet. You may comprise the song synthesizing | combining apparatus which synthesize | combines a song by accessing the data.

  (14) In the second embodiment, as the block is flicked, the pitch at the time of utterance of the phrase corresponding to the block is changed. However, you may change the utterance aspect of phrases other than a pitch. For example, as the block is flicked, the volume and playback speed of the phrase corresponding to the block may be changed.

  (15) In the third embodiment, when an operation of tilting a block is performed, a time change corresponding to the block tilt is given to the pitch when the phrase corresponding to the block is uttered. However, you may give the time change according to the inclination of a block to the utterance aspect of phrases other than a pitch. For example, a time change corresponding to the inclination of the block may be given to the volume and playback speed of the phrase corresponding to the block.

DESCRIPTION OF SYMBOLS 10 ... CPU, 20 ... Non-volatile memory, 21,21B ... Singing synthesis program, 100 ... Phrase data generation unit, 200,200A ... Singing synthesis unit, 300 ... Sequence data editing unit, 400 ... Sequence data generation unit, 25 ... Dictionary , 26 ... phrase database, 27 ... sequence database, 28 ... speech segment database, 30 ... ROM, 40 ... RAM, 50 ... display input unit, 60 ... sound system, 70 ... external interface group, 101 ... lyric analysis unit, 102 DESCRIPTION OF SYMBOLS ... Phonetic symbol generation part, 103 ... Phrase data composition part, 220 ... Reading control part, 230 ... Pitch conversion part, 240 ... Segment connection part, 250 ... Volume control part, 320 ... Pitch change control part, 301 ... Piano keyboard image 302 ... Sequence data editing area 303 ... Block layout area.

Claims (3)

Phrase data generating means for dividing the lyrics indicated by the lyric data into a plurality of fragments, generating a plurality of phrase data each indicating the utterance mode of the phrase of the synthesized singing sound from each fragment, and writing it into the storage means;
Display means;
Pointing position detecting means;
In response to a playback instruction for desired phrase data, the phrase data is read from the storage means, and a synthesized singing sound is generated using the read phrase data, each corresponding to a plurality of phrase data Phrase data corresponding to the block is displayed by displaying a plurality of blocks on the display means and detecting that one of the blocks displayed on the display means is designated by the designated position detecting means. When the synthetic singing sound is generated using the and the operation of tilting the block displayed on the display means is performed, when generating the synthetic singing sound according to the reproduction instruction of the phrase data corresponding to the block, A phrase synthesizer comprising: phrase reproducing means for giving a change corresponding to the inclination of the block to the synthesized singing sound . Phrase data generating means for dividing the lyrics indicated by the lyric data into a plurality of fragments, generating a plurality of phrase data each indicating the utterance mode of the phrase of the synthesized singing sound from each fragment, and writing it into the storage means;
Display means;
Pointing position detecting means;
In response to a playback instruction for desired phrase data, the phrase data is read from the storage means, and a synthesized singing sound is generated using the read phrase data, and a block layout area having a pitch axis is provided. Each block that is displayed on the display means and indicates each phrase data obtained from the lyrics data is displayed on the display means, and according to an operation performed by the user, a block corresponding to the desired phrase data is selected among these blocks. When the designated position detecting means detects that a block in the block layout area is laid out and the block in the block layout area is designated, a synthesized singing sound is generated using phrase data corresponding to the block, On the basis of the position of the block in the pitch axis direction And phrases reproducing means for controlling the pitch of 唱音
A singing voice synthesizing apparatus comprising: Phrase data generation process for dividing the lyrics indicated by the lyric data into a plurality of fragments, generating a plurality of phrase data each indicating the utterance mode of the phrase of the synthesized singing sound from each fragment, and writing to the storage means;
In response to a playback instruction for desired phrase data, the phrase data is read from the storage means, and a synthesized singing sound is generated using the read phrase data, each corresponding to a plurality of phrase data A plurality of blocks are displayed on the display means, and when the designated position detecting means detects that any one of the blocks displayed on the display means is designated, the phrase data corresponding to the block is used. When a synthetic singing sound is generated and an operation of tilting the block displayed on the display means is performed, when generating a synthetic singing sound in response to a phrase data reproduction instruction corresponding to the block, Phrase playback process that gives the synthesized singing sound a change corresponding to the inclination
A method for synthesizing a song.

Images