JP6413828B2 - Information processing method, information processing apparatus, and program - Google Patents

Description

  The present invention relates to a technical field of a method for generating meta information of a moving image for karaoke.

  Conventionally, video sites that distribute video data online are known. For example, the moving image site stores meta information in association with moving image data, and searches for meta information based on conditions specified by the user, thereby distributing moving image data desired by the user. The meta information is generated based on, for example, information input by the creator of the moving image data, or is generated by analyzing the moving image data. For example, Patent Literature 1 discloses a technique for generating meta information by extracting character information and information about image quality from a frame image included in moving image data.

JP 2010-020630 A

  In general, karaoke video data is generated from a plurality of material information such as video information, music sound information, and lyrics information. In this case, the video site registers and distributes the generated karaoke video data. When generating meta information of karaoke video data, it is necessary to analyze the karaoke video data. However, from the karaoke video data, detailed information included in the original material information is lost. For this reason, since it is difficult to accurately extract the original detailed information from the karaoke video data, there is a problem that meta information lacking accuracy is generated.

  The present invention has been made in view of the above points, and it is an object to provide an information processing method and the like that can easily generate accurate information as meta-information for searching for a karaoke video. To do.

The invention according to claim 1 is an information processing method executed by a computer of an information processing apparatus including temporary storage means, acquisition means, expansion means, karaoke video generation means, and meta information generation means. The acquisition means is a plurality of material information used as a material for a karaoke video, and includes video information, sound information indicating a performance sound of karaoke music, and lyrics information indicating lyrics of the karaoke music, wherein the karaoke music An acquisition step of acquiring a plurality of material information including at least lyrics information including an elapsed time from the start of the performance until the display of the lyrics is started, and the expanding means is acquired by the acquisition step An expansion step of expanding a plurality of material information once in the temporary storage means, and the karaoke video generation means is stored in the temporary storage means by the expansion step. A karaoke video generation step for generating the karaoke video based on the plurality of material information in a state in which the plurality of material information is expanded once, and the meta information generation means are the temporary storage means by the expansion step. Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once, and included in the lyrics information A meta information generating step of generating meta information including the elapsed time as a singing start time.

The invention according to claim 2 comprises temporary storage means, acquisition means, expansion means, karaoke video generation means, word extraction means, comparison means, difficulty determination means, and meta information generation means. An information processing method executed by a computer of an information processing device, wherein the acquisition means is a plurality of material information used as materials for a karaoke video, and includes video information, sound information indicating a performance sound of karaoke music, and the Acquiring step of acquiring a plurality of material information including at least lyrics information including display time from the start to the end of display of the word, for each word included in the lyrics, the lyrics information indicating the lyrics of the karaoke music The unfolding step in which the unfolding unit unfolds the plurality of material information acquired in the obtaining step in the temporary storage unit; and the karaoke video generation unit includes the unfolding step. A karaoke video generation step for generating the karaoke video based on the plurality of material information in a state where the plurality of material information is developed once in the temporary storage means by the step, and the word extraction means Among a plurality of high difficulty words stored in the first storage means for storing a high difficulty word predetermined as a word having a high difficulty and a reference time for singing the high difficulty word in association with each high difficulty word A word extraction step for extracting a high difficulty word included in the lyrics information developed in the temporary storage means; and the comparison means, for each high difficulty word extracted by the word extraction step, for the high difficulty word A comparison step for comparing the reference time of singing with the display time of the high difficulty word, and the difficulty level determination means determine the difficulty level of the lyrics based on the comparison result of the comparison step And at least one material information among the plurality of material information in a state in which the plurality of material information is expanded once in the temporary storage unit by the expanding step. A meta information generating step for generating meta information used for searching for the karaoke video, the meta information including the difficulty determined by the difficulty determining step.

The invention according to claim 3 is a temporary storage means and a plurality of material information used as a material of the karaoke video, and includes video information, sound information indicating the performance sound of the karaoke music, and lyrics indicating the lyrics of the karaoke music Acquisition means for acquiring a plurality of material information including at least lyrics information including elapsed time from the start of the performance of the karaoke music until the display of the lyrics is started, and acquired by the acquisition means A plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expansion means, and the plurality of material information is expanded once in the temporary storage means by the expansion means. And the karaoke video generation means for generating the karaoke video, and the expansion means in a state where the plurality of material information is expanded once in the temporary storage means. Meta information used for searching the karaoke video based on at least one material information of the plurality of material information, the meta information including the elapsed time included in the lyrics information as a singing start time. Meta information generating means for generating.
The invention according to claim 4 is a temporary storage means and a plurality of material information used as materials for a karaoke video, and includes video information, sound information indicating performance sound of karaoke music, and lyrics indicating lyrics of the karaoke music Acquisition means for acquiring a plurality of material information including at least lyrics information including display time from the start to the end of display of the word for each word included in the lyrics; and acquired by the acquisition means Further, based on the plurality of material information in a state in which the plurality of material information is expanded once in the temporary storage means by the expansion means, the expansion means that expands the plurality of material information once in the temporary storage means The karaoke video generation means for generating the karaoke video, a high difficulty word predetermined as a word having high singing difficulty, and a reference time for singing the high difficulty word for each high difficulty word Of the plurality of high difficulty words stored in the first storage means for storing in association, word extraction means for extracting high difficulty words included in the lyrics information developed in the temporary storage means; and the word extraction means Comparing means for comparing the reference time of singing the high difficulty word with the display time of the high difficulty word for each high difficulty word extracted by the above, difficulty of the lyrics based on the comparison result by the comparison means Based on at least one material information among the plurality of material information in a state in which the plurality of material information is expanded once in the temporary storage means by the expansion means, the difficulty level determination means for determining the degree, Meta-information used for searching for a karaoke video, and meta-information generating means for generating meta-information including the difficulty determined by the difficulty determining means.

According to a fifth aspect of the present invention, in a computer of an information processing apparatus including temporary storage means, a plurality of material information used as a material for a karaoke video, video information, sound information indicating a performance sound of karaoke music, and Acquiring a plurality of material information including at least lyrics information including lyrics time information indicating lyrics of a karaoke song and an elapsed time from the start of the performance of the karaoke song until the display of the lyrics is started. And a developing step of expanding the plurality of material information acquired in the acquiring step once in the temporary storage means, and a state in which the plurality of material information is expanded once in the temporary storage means by the expanding step The karaoke video generation step for generating the karaoke video based on the plurality of material information in Meta information used for searching for the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is developed once in the storage means, the lyrics information A meta information generating step of generating meta information including the elapsed time included in the singing start time as a singing start time.
According to a sixth aspect of the present invention, in a computer of an information processing apparatus including temporary storage means, a plurality of material information used as materials for a karaoke video, video information, sound information indicating a performance sound of karaoke music, and Acquiring step of acquiring a plurality of material information including at least lyrics information including display time from the start to the end of display of the word, for each word included in the lyrics, the lyrics information indicating the lyrics of the karaoke music In the state where the plurality of material information acquired by the acquisition step is expanded once in the temporary storage means, and the plurality of material information is expanded once in the temporary storage means by the expansion step A karaoke video generation step for generating the karaoke video based on the plurality of material information, and a predetermined high word as a singing difficulty level. Of the plurality of high difficulty words stored in the first storage means that stores the degree words and the reference time of singing the high difficulty words in association with each high difficulty word, the unfolded in the temporary storage means A word extraction step for extracting high difficulty words included in the lyrics information, a reference time for singing the high difficulty words, and a display time of the high difficulty words for each high difficulty word extracted by the word extraction step. The plurality of material information is expanded once in the temporary storage means by the comparing step for comparing, the difficulty determining step for determining the difficulty level of the lyrics based on the comparison result by the comparing step, and the expanding step. Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state of being determined by the difficulty determining step Characterized in that to execute, and meta information generation step of generating meta information including the difficulty.

According to the first to sixth aspects of the invention, a karaoke video is generated and meta information is generated based on the material information developed in the temporary storage means. Therefore, more accurate information can be easily generated as meta information for searching for this karaoke video as compared to the case where the karaoke video is generated from the karaoke video itself.

Furthermore , according to invention of Claim 1 , 3 or 5 , meta-information including exact song start time can be produced | generated easily.

Furthermore , according to invention of Claim 2 , 4 or 6 , meta information including the difficulty of a song can be produced | generated easily based on exact lyrics.

(A) is a figure showing an example of outline composition of communications system S of one embodiment. (B) is a figure showing an example of outline composition of karaoke animation generation server 1 of one embodiment. (A) is a figure which shows the structural example of telop data. (B) is a figure which shows the structural example of a difficult lyrics list. (C) is a figure which shows the structural example of a keyword list. (D) is a figure which shows the structural example of a kind list. (A) is a figure which shows the structural example of meta information containing singing start time. (B) is a figure which shows the structural example of meta information containing a singing difficulty level. (C) is a figure which shows the example of the appearance degree of a keyword. (D) is a figure which shows the structural example of the meta information containing a keyword. (E) is a figure which shows the structural example of the meta information containing a music classification. (A) is a flowchart showing an example of server processing in the karaoke video generation server 1. (B) is a flowchart showing an example of meta information generation processing in the karaoke video generation server 1. (A) is a flowchart which shows an example of the singing start time meta-information production | generation process in the karaoke video production | generation server 1, (B) is a flowchart which shows an example of the singing difficulty meta-information production | generation process in the karaoke animation production | generation server 1. FIG. . (A) is a flowchart showing an example of keyword meta information generation processing in the karaoke video generation server 1. (B) is a flowchart showing an example of music type meta information generation processing in the karaoke video generation server 1.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. Configuration of communication system]
FIG. 1A is a diagram illustrating a schematic configuration example of the communication system S of the present embodiment. As shown in FIG. 1A, the communication system S includes a karaoke video generation server 1, a data transmission terminal 2, a network storage 3, a karaoke video distribution server 4, and a plurality of user terminals 5. The karaoke video generation server 1 is an example of an information processing apparatus of the present invention. The karaoke video generation server 1, the data transmission terminal 2, the karaoke video distribution server 4, and the plurality of user terminals 5 are each connected to a network 10. The network 10 is configured by, for example, the Internet. The karaoke video generation server 1 and karaoke video distribution server 4 are connected to the network storage 3 via a network such as a LAN (Local Area Network).

  The karaoke video generation server 1 receives a plurality of material data from the data transmission terminal 2. And the karaoke moving image production | generation server 1 produces | generates karaoke moving image data based on several material data. The karaoke video data includes video data in which telops of lyrics of karaoke music are synthesized and performance sound data of karaoke music. The format of the karaoke video data may be MP4 (ISO / IEC 14496-14: 2003), for example.

  The material data is data serving as a material for generating karaoke video data. The plurality of material data includes at least video data, audio data, and telop data. The video data is data indicating video included in the karaoke video data. The format of the video data is, for example, H.264. H.264 or the like may be used. The audio data is data indicating the performance sound of karaoke music. The format of the audio data may be, for example, MIDI (Musical Instrument Digital Interface), AAC (Advanced Audio Coding), or the like. The telop data is data indicating the lyrics of karaoke music and the timing for displaying the lyrics. FIG. 2A is a diagram illustrating a configuration example of telop data. As shown in FIG. 2A, the telop data includes text information of lyrics. The telop data includes a display start time and a display end time for each character constituting the lyrics. The display start time indicates the time that elapses from when the performance of the karaoke music is started until the corresponding character is displayed. The display end time indicates the time that elapses from when the performance of the karaoke music is started until the display of the corresponding character is ended. In the example of FIG. 2A, the display start time and end time of “raw” are set to 500 milliseconds and 1000 milliseconds, respectively. The display start time and end time of “wheat” are set to 1000 milliseconds and 1200 milliseconds, respectively.

  Moreover, the karaoke moving image production | generation server 1 produces | generates the meta information used for the search of karaoke moving image data based on at least one among several material data. The karaoke video generation server 1 stores the generated karaoke video data and meta information in the network storage 3.

  The data transmission terminal 2 is used, for example, by an operator of a karaoke video data distribution service. For example, the operator inputs material data to the data transmission terminal 2 or operates the data transmission terminal 2 to create material data. The data transmission terminal 2 transmits the input or created material data to the karaoke video generation server 1.

  The network storage 3 stores a plurality of karaoke video data. The network storage 3 stores karaoke video data in association with identification information that can identify karaoke music. The identification information may be a music number, for example. The network storage 3 stores one or a plurality of meta information for each karaoke video data. The network storage 3 is composed of, for example, a hard disk drive.

  The karaoke video distribution server 4 searches the network storage 3 for meta information that matches the search condition transmitted from the user terminal 5. The karaoke video distribution server 4 identifies karaoke video data associated with the music number included in the searched meta information from the network storage 3. Thereby, the karaoke video delivery server 4 searches for karaoke video data. The karaoke video distribution server 4 distributes the searched karaoke video data to the user terminal 5 in a streaming manner.

  The user terminal 5 is used by a user of a karaoke distribution service. The user operates the user terminal 5 to specify a search condition for searching for karaoke video data. The user terminal 5 transmits the designated search condition to the karaoke video distribution server 4. Further, the user terminal 5 reproduces the karaoke video data distributed from the karaoke video distribution server 4. Examples of the user terminal 5 include a personal computer, a television, an STB, a mobile phone, a smartphone, a tablet computer, and a karaoke device.

[2. Configuration of Karaoke Movie Generation Server 1]
Next, the configuration of the karaoke video generation server 1 will be described with reference to FIGS. 1 (B) and 2 (B) to 2 (D). FIG. 1B is a diagram illustrating a schematic configuration example of the karaoke video generation server 1 of the present embodiment. As shown in FIG. 1B, the karaoke video generation server 1 includes a CPU (Center Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage unit 14, a communication unit 15, and the like. It is prepared for. These components are connected to the bus 16. The communication unit 15 is connected to the network 10. The storage unit 14 is configured by, for example, a hard disk drive. The storage unit 14 stores an OS, a server program, and the like. The server program is a program that causes a CPU as a computer to generate karaoke video data, meta information, and the like. The server program or the like may be downloaded from another server or the like via the network 10 or the like, for example. Alternatively, the server program or the like may be recorded on a recording medium such as an optical disc, a magnetic tape, or a memory card and read into the storage unit 14 via a drive device. The storage unit 14 stores information for generating meta information. Specifically, the storage unit 14 stores a difficult lyrics list, a keyword list, and a type list. These lists may be created, for example, by an operator of a distribution service for karaoke video data.

  FIG. 2B is a diagram illustrating a configuration example of the difficult lyrics list. The difficult lyrics list is a list of difficult lyrics of karaoke music. Difficult lyrics are words that are predetermined as words that are difficult to sing. Specifically, in the difficult lyrics list, the difficult lyrics and the standard singing time are registered in association with each difficult lyrics. The reference singing time is a reference value for the time when it is difficult to sing the corresponding difficult lyrics. In the example of FIG. 2B, the difficult lyrics “Namamugi” and the standard singing time of 1000 milliseconds are registered in association with each other. In this case, when the singing time of “Namamugi” is less than 1000 milliseconds in the karaoke music piece, it may be determined that the singing of “Namamugi” is difficult.

  FIG. 2C is a diagram illustrating a configuration example of the keyword list. The keyword list is a list indicating keywords and words related to the keywords. Specifically, in the keyword list, for each keyword, a keyword and a plurality of words related to the keyword are registered in association with each other. In the example of FIG. 2C, the words “cherry blossom”, “ume”, “graduation”, “nest”, and “enrollment” are registered in association with the keyword “spring”.

  FIG. 2D is a diagram illustrating a configuration example of the type list. The type list is a list of music types. Specifically, in the type list, for each type of music, a music type and one or a plurality of timbre information are registered in association with each other. The music type is identification information that can identify the type of music. The timbre information is information indicating the timbre of the sound source used for the performance of the type of music indicated by the music type. The timbre information assigning system may be the same as the timbre number assigning system defined by General MIDI, for example. In the type list shown in FIG. 2D, tone color information “shamisen” is registered in association with the music type “folk song”.

[3. Meta information generation]
For example, when generating meta information based on generated karaoke video data, it is necessary to analyze karaoke video data. However, the detailed information of the original material data is lost from the karaoke video data. Therefore, consider that the operator creates meta information based on at least one of a plurality of material data used for generating karaoke video data. However, if meta information is to be created later for completed karaoke video data, the following problem arises. That is, the material data used for generating the karaoke video data is stored in the storage means such as the data storage 3 and the storage unit 14 before the meta information is created, or the memory card, magnetic tape, optical disk, etc. It is necessary to save it on a recording medium. Therefore, computer resources are wasted to store material data. Moreover, since it is necessary to read material data from a memory | storage means or a recording medium, respectively at the time of the production | generation of karaoke moving image data, and the production | generation of meta information, it becomes a trouble twice.

  Therefore, the CPU 11 of the karaoke video generation server 1 develops a plurality of material data acquired from the data transmission terminal 2 in the RAM 13. The CPU 11 may expand a plurality of material data in the RAM 13 only once, for example. The CPU 11 generates karaoke video data based on a plurality of material data in a state in which a plurality of material data is expanded once in the RAM 13, and a plurality of materials in a state in which the plurality of material data is expanded in the RAM 13 once. Meta information is generated based on at least one of the data. That is, the CPU 11 generates meta information using exactly the material data used for generating the karaoke moving image data, or generates karaoke moving image data using the material data used for generating the meta information. To do. The RAM 13 is a working storage unit that temporarily stores data. The data stored in the RAM 13 will be erased sometime. While the meta information is stored in the RAM 13, the CPU 11 generates karaoke video data and meta information, thereby preventing waste of resources for storing the material data. The trouble of reading data can be reduced. Furthermore, accurate meta information can be generated based on the material data used for generating the karaoke video data. CPU11 may perform the production | generation of karaoke moving image data and the production | generation of meta information simultaneously or continuously, for example, and may produce | generate at a separate timing. Karaoke video data and meta information may be generated before the meta information is erased from the RAM 13. Further, the CPU 11 may generate karaoke video data first, or may generate meta information first. The RAM 13 is an example of a temporary storage unit of the present invention. The temporary storage means is not limited to the RAM 13. For example, if the material data is temporarily stored without the purpose of storage, the temporary storage means of the present invention can be applied to, for example, a nonvolatile memory.

  Next, a specific example of generation of meta information will be described with reference to FIG. For example, the CPU 11 may generate meta information including the singing start time. The singing start time is the time from the start of the performance of the karaoke music to the start of the singing of the first lyrics of the karaoke music. Specifically, the CPU 11 acquires the display start time of the first character of the lyrics from the telop data in the RAM 13 as the singing start time. When reproducing the karaoke video data, the user sings in accordance with the display timing of the lyrics telop. Therefore, the display start time of the first lyrics portion can be considered as the singing start time. FIG. 3A is a diagram illustrating a configuration example of meta information including a singing start time. The meta information shown in FIG. 3A includes a music number and a singing start time. When searching for karaoke video data, the user can specify a singing start time condition as a search condition, for example. As this condition, for example, designation of 10 seconds, 5 seconds or more, 20 seconds or less, etc. may be possible. The karaoke video distribution server 4 searches the network storage 3 for meta information including the singing start time that satisfies the condition of the specified singing start time.

  Further, for example, the CPU 11 may generate meta information including the singing difficulty level. The singing difficulty level is information indicating the difficulty of singing karaoke music. The higher the singing difficulty, the more difficult it is to sing. Specifically, the CPU 11 extracts difficult lyrics included in the telop data from the difficult lyrics registered in the difficult lyrics list. CPU11 acquires the singing time of the extracted difficult lyrics from telop data. For example, the CPU 11 acquires the time from the display start time of the first character of difficult lyrics to the display end time of the last character of difficult lyrics as the singing time. The CPU 11 compares the singing time acquired from the telop data with the reference singing time for difficult lyrics. And CPU11 determines a singing difficulty level based on the result of this comparison. For example, CPU11 may determine a high singing difficulty degree, so that there are many difficult lyrics whose acquired singing time is shorter than reference | standard singing time. For example, in the telop data shown in FIG. 2A, “raw” and “wheat” are continuous. Therefore, the telop data includes the difficult lyrics “raw wheat” shown in FIG. The display start time of “raw” is 500 milliseconds, and the display end time of “wheat” is 1200 milliseconds. Therefore, the singing time of “Namamugi” is 700 milliseconds. Since the standard singing time of the difficult lyrics “Namamugi” shown in FIG. 2B is 1000 milliseconds, the CPU 11 may increase the singing difficulty level by 1, for example. FIG. 3B is a diagram illustrating a configuration example of meta information including the singing difficulty level. The meta information shown in FIG. 3B includes a music number and a singing difficulty level. When searching for karaoke video data, the user can specify a singing difficulty level condition as a search condition, for example. As this condition, for example, designation of difficulty 0, difficulty 5 or higher, difficulty 8 or lower, etc. may be possible. The karaoke video distribution server 4 searches the network storage 3 for meta information including the singing difficulty level that satisfies the specified singing difficulty level.

  Further, for example, the CPU 11 may generate meta information including a keyword. Specifically, the CPU 11 extracts, from the keyword list, a keyword associated with a word included in the telop data among words registered in the keyword list. CPU11 counts for every keyword registered into the keyword list by making the extraction frequency of the extracted keyword into appearance degree. FIG. 3C is a diagram illustrating an example of the appearance degree of keywords. In the example of FIG. 3C, the appearance degree of the keyword “Spring” is 4, and the appearance degree of the keyword “Enka” is 1. CPU11 determines the keyword whose appearance degree exceeds predetermined frequency as a keyword of karaoke music. FIG. 3D is a diagram illustrating a configuration example of meta information including a keyword. The meta information shown in FIG. 3D includes a music number and one or more keywords. For example, when the predetermined frequency is 3, “Spring” is determined as the keyword of the karaoke song among the keywords shown in FIG. The predetermined frequency may be other than 3. When searching for karaoke video data, the user can specify a keyword as a search condition, for example. The karaoke video distribution server 4 searches the network storage 3 for meta information including the designated keyword.

  Assume that the meta information shown in FIGS. 3A, 3B, and 3D is generated based on karaoke video data. In this case, it is necessary to analyze the video data included in the karaoke video data and extract the lyrics. As a result, the processing load on the CPU 11 increases and the lyrics may not be extracted accurately. In contrast, in the present embodiment, since the CPU 11 generates meta information based on telop data that clearly indicates lyrics, accurate meta information can be generated.

  Further, for example, the CPU 11 may generate meta information including a music type. For example, when the audio data is MIDI data, the audio data includes one or more tracks. A track may contain timbre information. This timbre information indicates the timbre of the sound source used for the performance of karaoke music. The CPU 11 extracts, from the type list, the music type associated with the timbre information included in the audio data from the timbre information registered in the type list. Then, the CPU 11 generates meta information including the extracted music type. FIG. 3E is a diagram illustrating a configuration example of meta information including a music type. The meta information shown in FIG. 3E includes a music number and one or more music types. When searching for karaoke video data, the user can specify a music type as a search condition, for example. The karaoke video distribution server 4 searches the network storage 3 for meta information including the designated music type. In addition, about one karaoke music, a some music type may be extracted. In this case, for example, the CPU 11 may determine the music type extracted with the highest frequency and generate meta information including only the determined music type. It is considered that the more frequently extracted music types have a higher probability of indicating the type of karaoke music. Alternatively, for example, the CPU 11 may determine a plurality of song types up to a predetermined number in order from the extracted song type having the highest frequency, and generate meta information including the determined plurality of song types. Alternatively, the CPU 11 may generate information indicating the extracted frequency for each extracted music type. Then, the CPU 11 may generate meta information including the music type and information indicating the extracted frequency.

  Suppose that the meta information shown in FIG. 3E is generated based on karaoke video data. In this case, it is necessary to analyze the sound data included in the karaoke video data and specify the sound source and tone color. However, this audio data is data obtained by sampling audio, such as AAC. For this reason, the processing load on the CPU 11 increases and the timbre may not be specified accurately. In contrast, in the present embodiment, since the CPU 11 generates meta information based on MIDI format audio data including timbre information, accurate meta information can be generated.

  The CPU 11 may generate all of the four types of meta information described so far, or may generate only a part of the meta information. Further, the CPU 11 may generate another type of meta information, for example. Further, the CPU 11 may generate meta information based on video data included in a plurality of material data. Further, the CPU 11 may generate meta information based on, for example, two or more material data. Further, the CPU 11 may generate meta information including a plurality of types of information. For example, the CPU 11 may generate one piece of meta information including a singing start time, a difficulty level, a keyword, a music type, and the like.

[4. Operation of communication system S]
Next, the operation of the communication system S of the present embodiment will be described with reference to FIGS. FIG. 4A is a flowchart illustrating an example of server processing in the karaoke video generation server 1. For example, when a server program is activated in the karaoke video generation server 1, the CPU 11 executes server processing. As shown in FIG. 4A, the CPU 11 determines whether or not the server program ends (step S1). At this time, if the CPU 11 determines that the server program does not end (step S1: NO), the CPU 11 proceeds to step S2. In step S <b> 2, the CPU 11 determines whether or not a plurality of material data has been received from the data transmission terminal 2. At this time, if the CPU 11 determines that a plurality of material data has not been received (step S2: NO), the process proceeds to step S1. On the other hand, if the CPU 11 determines that a plurality of material data has been received (step S2: YES), the CPU 11 proceeds to step S3.

  In step S <b> 3, the CPU 11 expands the received plurality of material data in the RAM 13. Moreover, CPU11 acquires the music number of the karaoke moving image data produced | generated. For example, the CPU 11 may acquire a music number from the data transmission terminal 2, or the CPU 11 may generate a music number. Next, the CPU 11 executes meta information generation processing to be described later (step S4).

  Next, the CPU 11 generates karaoke video data based on the plurality of material data expanded in the RAM 13 (step S5). For example, when the audio data included in the material data is MIDI format data, the CPU 11 converts the audio data into audio data in a format such as AAC, for example. Further, the CPU 11 synthesizes the lyrics indicated by the telop data included in the material data with the video data included in the material data. For example, for each character constituting the lyrics, the CPU 11 starts displaying the character at the timing of the display start time from the start of the video reproduction, and ends the display of the character at the timing of the display end time from the start of the video reproduction. Thus, the synthesis is performed. The CPU 11 multiplexes the audio data and the video data synthesized with the lyrics to generate karaoke video data. Next, the CPU 11 stores the karaoke video data and the music number in the network storage 3 in association with each other. Then, the CPU 11 proceeds to step S1. In step S1, if the CPU 11 determines that the server program ends (step S1: YES), the CPU 11 ends the server process.

  FIG. 4B is a flowchart illustrating an example of meta information generation processing in the karaoke video generation server 1. As shown in FIG. 4B, the CPU 11 performs singing start time meta information generation processing (step S11), singing difficulty meta information generation processing (step S12), keyword meta information generation processing (step S13), and The music type meta information generation process (step S14) is executed to end the meta information generation process.

  FIG. 5A is a flowchart illustrating an example of the singing start time meta information generation process in the karaoke video generation server 1. As shown in FIG. 5A, the CPU 11 acquires the display start time of the first character of the lyrics as the singing start time from the telop data expanded in the RAM 13 (step S21). Next, the CPU 11 generates meta information including the music number and the acquired singing start time. And CPU11 memorize | stores meta information in the network storage 3 (step S22), and complete | finishes a song start time meta-information production | generation process.

  FIG. 5B is a flowchart showing an example of the singing difficulty level meta information generation processing in the karaoke video generation server 1. As shown in FIG. 5B, the CPU 11 reads the difficult lyrics list from the storage unit 14 (step S31). Next, the CPU 11 sets the singing difficulty level to 0 and sets the number i to 1 (step S32). Next, the CPU 11 determines whether or not the difficult lyric (i) is included in the telop data expanded in the RAM 13 (step S33). Difficult lyrics (i) is the i-th difficult lyrics among the difficult lyrics included in the difficult lyrics list. When the CPU 11 determines that the difficult text (i) is not included in the telop data (step S33: NO), the CPU 11 proceeds to step S37. On the other hand, if the CPU 11 determines that the difficult text (i) is included in the telop data (step S33: YES), the CPU 11 proceeds to step S34.

  In step S34, the CPU 11 obtains the display start time of the first character of the difficult lyrics (i) and the display end time of the last character of the difficult lyrics (i) from the telop data. And CPU11 acquires singing time by subtracting display start time from acquired display end time. Next, the CPU 11 determines whether or not the reference singing time registered in the difficult lyric list in association with the difficult lyric (i) is longer than the acquired singing time (step S35). At this time, if the CPU 11 determines that the reference singing time is not longer than the acquired singing time (step S35: NO), the CPU 11 proceeds to step S37. On the other hand, if the CPU 11 determines that the reference singing time is longer than the acquired singing time (step S35: YES), the CPU 11 proceeds to step S36. In step S36, the CPU 11 increases the singing difficulty level by one.

  Next, the CPU 11 determines whether or not the number i is less than the number of difficult lyrics registered in the difficult lyrics list (step S37). At this time, if the CPU 11 determines that the number i is less than the number of difficult lyrics (step S37: YES), the CPU 11 proceeds to step S38. In step S38, the CPU 11 increments the number i by 1 and proceeds to step S33. On the other hand, if the CPU 11 determines that the number i is not less than the number of difficult lyrics (step S37: NO), the CPU 11 proceeds to step S39. In step S39, the CPU 11 generates meta information including the music number and the singing difficulty level. And CPU11 memorize | stores meta information in the network storage 3, and complete | finishes a singing difficulty meta information production | generation process.

  FIG. 6A is a flowchart showing an example of keyword meta information generation processing in the karaoke video generation server 1. As shown in FIG. 6A, the CPU 11 reads a keyword list from the storage unit 14 (step S41). Next, the CPU 11 sets the appearance degree of each keyword included in the keyword list to 0. Further, the CPU 11 sets the number i to 1 (step S42). Next, the CPU 11 determines whether or not the word (i) is included in the telop data expanded in the RAM 13 (step S43). The word (i) is the i-th word among the words included in the keyword list. If the CPU 11 determines that the word (i) is not included in the telop data (step S43: NO), the CPU 11 proceeds to step S45. On the other hand, when the CPU 11 determines that the word (i) is included in the telop data (step S43: YES), the CPU 11 proceeds to step S44. In step S44, the CPU 11 increases the appearance degree of the keyword registered in the keyword list in association with the word (i) by 1, and proceeds to step S45.

  In step S45, the CPU 11 determines whether the number i is less than the number of words registered in the keyword list. At this time, if the CPU 11 determines that the number i is less than the number of words (step S45: YES), the CPU 11 proceeds to step S46. In step S46, the CPU 11 increments the number i by 1 and proceeds to step S43. On the other hand, when the CPU 11 determines that the number i is not less than the number of words (step S45: NO), the CPU 11 proceeds to step S47.

  In step S47, the CPU 11 sets the number i to 1. Next, the CPU 11 determines whether or not the appearance degree of the keyword (i) is greater than 3 (step S48). Keyword (i) is the i-th keyword among the keywords registered in the keyword list. When the CPU 11 determines that the appearance degree of the keyword (i) is not greater than 3 (step S48: NO), the CPU 11 proceeds to step S50. On the other hand, if the CPU 11 determines that the appearance degree of the keyword (i) is greater than 3 (step S48: YES), the CPU 11 proceeds to step S49. In step S49, the CPU 11 stores the keyword (i) in the RAM 13 as one of the keywords of the karaoke piece. Next, the CPU 11 proceeds to step S50.

  In step S50, the CPU 11 determines whether the number i is less than the number of keywords registered in the keyword list. At this time, if the CPU 11 determines that the number i is less than the number of keywords (step S50: YES), the CPU 11 proceeds to step S51. In step S51, the CPU 11 increments the number i by 1 and proceeds to step S48. On the other hand, if the CPU 11 determines that the number i is not less than the number of keywords (step S50: NO), the CPU 11 proceeds to step S52. In step S <b> 52, the CPU 11 generates meta information including a music number and a keyword stored in the RAM 13. Then, the CPU 11 stores the meta information in the network storage 3 and ends the keyword meta information generation process.

  FIG. 6B is a flowchart illustrating an example of music type meta information generation processing in the karaoke video generation server 1. As shown in FIG. 6B, the CPU 11 reads the type list from the storage unit 14 (step S61). Next, the CPU 11 sets the number i to 1 (step S62). Next, the CPU 11 determines whether or not the timbre information (i) is included in the audio data expanded in the RAM 13 (step S63). The timbre information (i) is the i-th timbre information among the timbre information included in the type list. When the CPU 11 determines that the tone color information (i) is not included in the telop data (step S63: NO), the CPU 11 proceeds to step S65. On the other hand, if the CPU 11 determines that the telop data includes the timbre information (i) (step S63: YES), the CPU 11 proceeds to step S64. In step S64, the CPU 11 stores the song type registered in the type list in association with the timbre information (i) in the RAM 13 as a song type that may indicate the type of karaoke song. Next, the CPU 11 proceeds to step S65.

  In step S65, the CPU 11 determines whether the number i is less than the number of timbre information registered in the type list. At this time, if the CPU 11 determines that the number i is less than the number of timbre information (step S65: YES), the CPU 11 proceeds to step S66. In step S66, the CPU 11 increments the number i by 1 and proceeds to step S63. On the other hand, if the CPU 11 determines that the number i is not less than the number of timbre information (step S65: NO), the CPU 11 proceeds to step S67. In step S <b> 67, the CPU 11 generates meta information including the music number and the music type stored in the RAM 13. Then, the CPU 11 stores the meta information in the network storage 3 and ends the music type meta information generation process.

  As described above, according to the present embodiment, the karaoke video generation server 1 develops a plurality of material information acquired from the data transmission terminal 2 in the RAM 13. Further, the karaoke video generation server 1 generates karaoke video data based on a plurality of material information expanded in the RAM 13. Further, the karaoke video generation server 1 generates meta information based on at least one material information among a plurality of material information expanded in the RAM 13. Therefore, it is possible to easily generate more accurate meta information as compared with the case of generating from the karaoke video itself.

DESCRIPTION OF SYMBOLS 1 Karaoke moving image production | generation server 2 Data transmission terminal 3 Network storage 4 Karaoke moving image delivery server 5 User terminal 10 Network 11 CPU
12 ROM
13 RAM
14 storage unit 15 communication unit S communication system

Claims (6)

An information processing method executed by a computer of an information processing apparatus including temporary storage means, acquisition means, expansion means, karaoke video generation means, and meta information generation means,
The acquisition means is a plurality of material information used as a material of a karaoke video, and is video information, sound information indicating a performance sound of karaoke music, and lyrics information indicating lyrics of the karaoke music, An acquisition step of acquiring a plurality of material information including at least lyrics information including an elapsed time from the start of performance until the display of the lyrics is started;
The unfolding step in which the unfolding means unfolds the plurality of material information acquired in the acquiring step once in the temporary storage unit;
A karaoke video generation step, wherein the karaoke video generation means generates the karaoke video based on the plurality of material information in a state in which the plurality of material information is once expanded in the temporary storage means by the expansion step; ,
The meta information generating unit is configured to generate the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage unit by the expanding step. Meta information generation step for generating meta information that is meta information used for search and includes the elapsed time included in the lyrics information as a singing start time;
An information processing method comprising: Executed by a computer of an information processing apparatus comprising temporary storage means, acquisition means, expansion means, karaoke video generation means, word extraction means, comparison means, difficulty determination means, and meta information generation means. An information processing method,
The acquisition means is a plurality of material information used as a material of a karaoke video, and includes video information, sound information indicating a performance sound of karaoke music, and lyrics information indicating lyrics of the karaoke music, and is included in the lyrics An acquisition step of acquiring a plurality of material information including at least lyrics information including a display time from the start to the end of the display of each word,
The unfolding step in which the unfolding means unfolds the plurality of material information acquired in the acquiring step once in the temporary storage unit;
A karaoke video generation step, wherein the karaoke video generation means generates the karaoke video based on the plurality of material information in a state in which the plurality of material information is once expanded in the temporary storage means by the expansion step; ,
The word extraction means is stored in a first storage means for storing a high difficulty word predetermined as a word having a high difficulty in singing and a reference time for singing the high difficulty word in association with each high difficulty word. A word extraction step of extracting a high difficulty word included in the lyrics information developed in the temporary storage means among the plurality of high difficulty words;
A comparison step in which the comparison means compares, for each high difficulty word extracted by the word extraction step, a reference time of singing the high difficulty word and a display time of the high difficulty word;
The difficulty level determination means determines the difficulty level of the lyrics based on the comparison result of the comparison step; and
The meta information generating unit is configured to generate the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage unit by the expanding step. Meta information used for the search, and a meta information generation step for generating meta information including the difficulty level determined by the difficulty level determination step;
An information processing method comprising: Temporary storage means;
A plurality of material information used as a material for the karaoke video, the video information, the sound information indicating the performance sound of the karaoke music, and the lyric information indicating the lyrics of the karaoke music, and the performance of the karaoke music is started. Acquisition means for acquiring a plurality of material information including at least lyrics information including an elapsed time until display of the lyrics is started;
Expanding means for expanding the plurality of material information acquired by the acquiring means once in the temporary storage means;
Karaoke video generation means for generating the karaoke video based on the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expansion means;
Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage unit by the expanding unit. Meta information generating means for generating meta information including the elapsed time included in the lyrics information as a singing start time;
An information processing apparatus comprising: Temporary storage means;
It is a plurality of material information that becomes the material of the karaoke video, video information, sound information indicating the performance sound of the karaoke music, and lyric information indicating the lyrics of the karaoke music, for each word included in the lyrics, Acquisition means for acquiring a plurality of material information including at least lyrics information including a display time from the start to the end of display of the word;
Expanding means for expanding the plurality of material information acquired by the acquiring means once in the temporary storage means;
Karaoke video generation means for generating the karaoke video based on the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expansion means;
A plurality of high difficulty words stored in a first storage means for storing a high difficulty word predetermined as a word having a high difficulty of singing and a reference time of singing the high difficulty word in association with each high difficulty word Among them, word extraction means for extracting high difficulty words included in the lyrics information developed in the temporary storage means,
For each high difficulty word extracted by the word extraction means, a comparison means for comparing a reference time of singing the high difficulty word and a display time of the high difficulty word;
Difficulty determination means for determining the difficulty level of the lyrics based on the comparison result by the comparison means;
Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage unit by the expanding unit. Meta information generation means for generating meta information including the difficulty level determined by the difficulty level determination means;
An information processing apparatus comprising: In the computer of the information processing apparatus provided with temporary storage means,
A plurality of material information used as a material for the karaoke video, the video information, the sound information indicating the performance sound of the karaoke music, and the lyric information indicating the lyrics of the karaoke music, and the performance of the karaoke music is started. An acquisition step of acquiring a plurality of material information including at least lyrics information including an elapsed time until display of the lyrics is started;
An unfolding step of unfolding the plurality of material information obtained by the obtaining step once in the temporary storage means;
A karaoke video generation step for generating the karaoke video based on the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expansion step;
Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expanding step. A meta information generating step for generating meta information including the elapsed time included in the lyrics information as a singing start time;
A program characterized by having executed. In the computer of the information processing apparatus provided with temporary storage means,
It is a plurality of material information that becomes the material of the karaoke video, video information, sound information indicating the performance sound of the karaoke music, and lyric information indicating the lyrics of the karaoke music, for each word included in the lyrics, An acquisition step of acquiring a plurality of material information including at least lyrics information including a display time from the start to the end of display of the word;
An unfolding step of unfolding the plurality of material information obtained by the obtaining step once in the temporary storage means;
A karaoke video generation step for generating the karaoke video based on the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expansion step;
A plurality of high difficulty words stored in a first storage means for storing a high difficulty word predetermined as a word having a high difficulty of singing and a reference time of singing the high difficulty word in association with each high difficulty word A word extraction step of extracting high difficulty words included in the lyrics information developed in the temporary storage means,
For each high difficulty word extracted by the word extraction step, a comparison step for comparing a reference time of singing the high difficulty word and a display time of the high difficulty word;
A difficulty determination step for determining the difficulty level of the lyrics based on the comparison result of the comparison step;
Meta information used for searching the karaoke video based on at least one material information among the plurality of material information in a state where the plurality of material information is expanded once in the temporary storage means by the expanding step. A meta information generation step for generating meta information including the difficulty level determined by the difficulty level determination step;
A program characterized by having executed.

Images