WO2017182533A1

WO2017182533A1 - Method and system for playing musical instruments

Info

Publication number: WO2017182533A1
Application number: PCT/EP2017/059309
Authority: WO
Inventors: Tim VETS
Original assignee: Universiteit Gent
Priority date: 2016-04-19
Filing date: 2017-04-19
Publication date: 2017-10-26

Abstract

The present invention relates to a system for playing a plucked string instrument, the system comprising a plectrum, for stroking strings, provided with two electrically separated electrodes and a body, one electrode being positioned at one side of the body, the other electrode being position at the opposite side of the body, the system further comprising a plucked string instrument wherein at least two strings are electrically insulated from each other. Either the electrodes of the plectrum are connected to a voltage source, while the at least two strings of the plucked string instrument are connected to an electrical detector, or the at least two strings of the plucked string instrument are connected to a voltage source while the electrodes of the plectrum are electrically connected to a voltage source. In both cases, the system can detect electrical contact between one of the two electrodes and one of the strings.

Description

Method and system for playing musical instruments

Field of the invention

The invention relates to the field of musical instruments. More specifically it relates to an augmentation system for playing musical instruments and use thereof. Background of the invention

In the field of music and musical instruments, the advent of computers capable of processing audio with low latency has stimulated research on augmented musical instruments. Augmented musical instruments are traditional musical instruments onto which additional technology (hardware and software) is added so that the musical instrumental playing, and/or its sonic output reaches sound processing beyond what is normally possible on the traditional musical instrument.

In this context, musicians are confronted with the question of how augmentation affects payability and expressiveness. Often, augmentation requires a new playing technique in order to control the expressive possibilities offered by added electronics. Augmentation may affect the control of expressiveness at the micro-timing level. By adding electronics, control may appear less direct, less based on haptic feedback and physical causation. The musical end result of research on augmented musical instruments can benefit from a better integration of the musical instrument's electronic sound processing with the musician's sensorimotor control during performance.

A notable example of augmentation is the use of pitch tracking allowing the guitar to control Musical Instrument Digital Interface (MIDI) capable sound sources. Commercial MIDI-guitar products are aimed at tracking the played pitches and return a symbolic representation of played notes in the form of MIDI note, velocity and possibly pitchbend or modulation values. With optimised tracking algorithms, some of these applications obtain good results even when applied to the monaural audio signal of the guitar that contains a combination of all six strings. Some MIDI guitar systems have employed ultrasound technologies to achieve this goal. A hexaphonic pickup is often used for MIDI guitar applications, in order to split the polyphonic signal to be tracked into six monophonic signals. The aforementioned applications track played notes, but do not track the activity of the player.

The potential of augmented guitar picks (or plectrums) has been explored in the past by some academic as well as commercial projects. In the document "The midi pick: trigger serial data, samples, and midi from a guitar pick, by Vanegas, . In Proceedings of the 7th international conference on New interfaces for musical expression, pages 330-332. ACM, 2007", a system is described in which a guitar pick comprises a pressure sensor, providing the player with a continuous controller. The controller has to be used by varying the pressure of the fingers on the guitar pick. This requires the player to use gestures that are not part of conventional guitar playing, which may introduce undesirable distraction from the performance.

Summary of the invention

It is an object of embodiments of the present invention to provide a system for playing musical instruments or assisting therein, as well as use thereof. The method and system may be suitable for a large plurality of musical instruments such as for example bowed or plucked string instruments, , percussion instruments, wind instruments such as woodwind instruments or brass instruments, keyboard instruments, lamellophones (e.g. kalimba), quintephones (e.g. theremin) etc. .

It is an advantage of embodiments of the present invention that sound systems are provided allowing at least one of sound manipulation and motion data storage, improving learning or playing musical instruments, allowing objective technique analysis, and allowing fluent play and prediction.

It is an advantage of embodiments of the present invention that prediction of a physical interaction with a musical instrument, such as for example prediction of the probability of an onset, prediction of the intensity of a physical interaction, etc. can be performed. The information obtained by predicting within a range of milliseconds ahead of the actual playing can be used for predictive sound processing, also referred to as "pre-ambles" and accurate expressive control, e.g. based on probabilities of an onset.

It is an advantage of embodiments of the present invention that the system allows distinguishing over different playing features such as up-strokes versus down-strokes, string selections, the continuous velocity of gestures, fingerings and note selections. This allows to explore new expressive possibilities.

It is an advantage of embodiments of the present invention that the system provides a learning solution when playing a musical instrument, such as for example a string-based plucked instrument such as a guitar, easily adaptable to any plucked string instrument, string instrument more generally, percussion instrument, wind instrument, keyboard instrument, lamellophone, quintephone, etc. It is an advantage of embodiments of the present invention that, for the case of a plucked string instrument or string instrument, the system is not dependent on the pressure of fingers on the guitar pick. The system therefore is independent of a particular pressure that the user applies, rendering the system more easily applicable for various guitar players. Again this is extendable to any plucked string instrument.

According to one aspect, the present invention relates to a system for playing an musical instrument, the system comprising a detection system for detecting sensorimotor preparation for physical interaction of a player with the musical instrument and a controller for receiving data signals from the detector and for predicting a physical interaction with the musical instrument prior to said physical interaction with the musical instrument based on the received data signals, said predicting being performed using reservoir computing.

Said predicting may be performed using a recursive neural network.

Said predicting a physical interaction may comprise predicting any or a combination of an onset, a location of physical interaction with the musical instrument, an intensity of physical interaction with the musical instrument, an articulation variation or a timbral variation, pitch choices in melodic or harmonic playing and/or their intervallic relationship to other pitch choices, rhythmic note values in relation to the meter and tempo of the current musical context, data collection for statistical analysis and community interactions or educational purposes.

The system may comprise furthermore a module for further audio processing based on a predicted physical interaction obtained using the controller.

The controller furthermore may be configured for inducing an augmentation of the sound produced by the musical instrument based on the predicted physical interaction with the musical instrument.

The controller furthermore may be configured for providing feedback to the player for training, direction or conducting purposes.

The detection system for detecting sensorimotor preparation for physical interaction with the musical instrument may be a detection system for detecting movement of the player with respect to the musical instrument.

The detection system may comprise one or more of an optical camera or infrared camera, a radar, an ultrasound detector, a vibration sensor, a gyroscope, an accelerometer, a piezo sensor or a hall effect sensor. The system furthermore may be adapted for providing haptic feedback to the player. In one embodiment, a vibrating element may be introduced in the musical instrument or a part thereof or a device used for playing the musical instrument, such as for example a plectrum or a drumstick, and the system may be adapted for inducing a vibration in the vibrating element for providing haptic feedback to the player. This may be used as a cueing system or as a didactical device, giving playing guidance or cues to the player without intruding on the auditory experience of the player (as opposed to e.g. in-ear cueing systems). Additional applications may consist in providing haptic feedback for virtual musical instruments, e.g. A virtual guitar's interface may consist of a touch-sensitive surface or other interface without physical strings. By coupling the playing of these virtual strings with haptic feedback upon pick- to-virtual-string collisions, the player can be given a more realistic experience and hence more accurate control.

The musical instrument may be any of a string based plucking instrument, a string instrument, a percussion instrument, a wind instrument, keyboard instrument, lamellophone, quintephone, etc..

The system may be especially suitable for use with (or comprising) a plectrum and musical instrument as described in other aspects of the present invention. The detection system for detecting sensorimotor preparation for physical interaction of a player with the musical instrument may then be integrated in the plectrum. The detector system may comprise a piezo sensor and/or an accelerometer. The controller may be adapted for deriving whether a stroke on the string or strings will be an up-stroke or a down-stroke, and/or for deriving which string will be stroke.

The present invention also relates in a second aspect to the use of a system as described in the first aspect for sound manipulation.

The present invention furthermore relates in a third aspect to the use of a system as described in the first aspect for training, directing or conducting purposes.

The present invention also relates in a fourth aspect to the use of a system as described in the first aspect for prediction of a stroke on the plucked string instrument. According to a fifth aspect, the present invention not being limited thereto, a system for playing a plucked string instrument is disclosed, the system comprising

- a plectrum for stroking strings, the plectrum being provided with two electrically separated electrodes and a plectrum body, one electrode being positioned at one side of the plectrum body, the other electrode being position at the opposite side of the plectrum body, the electrodes being preferably separated by the plectrum body along its thickness, the plectrum body preferably comprising non-conductive materials, so electrical isolation is provided between the electrodes, the system further comprising:

- a plucked string instrument wherein at least two strings are electrically insulated from each other,

wherein either the electrodes of the plectrum are connected to a voltage source while the at least two strings of the plucked string instrument are connected to an electrical detector for detecting an electrical contact between one of the two electrodes and one of the strings, or the at least two strings of the plucked string instrument are connected to a voltage source while the electrodes of the plectrum are electrically connected to a voltage source, for detecting an electrical contact between one of the two electrodes and one of the strings. The plectrum furthermore comprises the detection system for detecting sensorimotor preparation for physical interaction of a player with the musical instrument.

The at least 2 strings may be only 2 strings, a plurality of the strings or all of the strings present in the plucked string instrument.

The plucked string instrument may comprise a plectrum comprising a striking zone, used to strike the strings, and a gripping zone, used to hold the plectrum. The striking zone of the plectrum preferably comprises the two electrodes.

The plucked string instrument may for example be any of a veena, banjo, ukulele, guitar, harp, lute, mandolin, oud or sitar.

In some embodiments of the present invention the electrodes of the plectrum are connected to an electrical detector, the at least two strings of the plucked string instrument are electrically connected to at least one voltage source and put at different voltages, so that electrical contact between one of the electrodes of the plectrum and a particular string can be distinguished using the electrical detector based on a different electrical response. It is an advantage of embodiments of the present invention that a single electrical detector allows distinguishing which strings are touched and whether these strings are touched with an up or down stroke. In some embodiments of the present invention, the two separated electrodes of the plectrum are configured as normally-open contacts. This allows to measure touching the guitar strings by identifying a voltage when the electrode is contacted with the strings.

The plucked instrument may comprise a system for electrically insulating the at least two strings and preferably all of the strings. The plucked string instrument may be an existing, conventional musical instrument, adapted for providing electrical insulation between its at least two strings. For example, the plucked string instrument comprises an anchoring bar for anchoring the strings, wherein the anchoring bar is made of electrically insulating material. Bridges and other features may be included, for example an intermediate bar pushing the strings down to create tension to keep the strings on bridge saddles, wherein the intermediate bar is made of electrically insulating material. In some embodiments of the present invention, the bridge saddles provide further isolation. For example, the bridge saddles may be isolating. For example, the bridge saddles are isolated from the strings by a thin isolating layer. The thin isolating layer may for example be a thin plastic layer. It is an advantage of embodiments of the present invention that the layer isolating the bridge saddles from the strings can be thin so that they have a negligible influence on the vibration of the strings. It is an advantage of embodiments of the present invention that the thin isolating layer can be easily removed and installed, if for example the system is to be added to, or removed from, a standard musical instrument such as a guitar.

In some embodiments of the present invention, the strings of the musical instrument are electrically connected to conduction wires using cylindrical screw clamps. Alternatively, the strings of the plucked string instrument are electrically connected to conduction wires using clamps allowing to insert the strings from a side of the clamp. It is an advantage of some embodiments of the present invention that the system can be easily installed on and removed from a guitar, e.g. without the need for removing the strings from the guitar. This further allows customization of a standard musical instrument.

In some embodiments of the present invention, the electrodes of the plectrum are copper plates attached to both opposite sides or surfaces of the plectrum body.

A system according to embodiments of the present invention may further comprise a processing unit. The connection between the plectrum and the processing unit may be made by electrical wires. It is an advantage of embodiments of the present invention that no latency is introduced by the connection between the guitar pick and the processing unit. Alternatively, the connection between the plectrum and the processing unit may be based on wireless communication, advantageously reducing the amount of wiring and improving comfort. In embodiments of the present invention, the plectrum further comprises a piezo sensor, which advantageously allows detecting variables such as stroke force, vibrations, etc. Alternatively or additionally, the plectrum further comprises an accelerometer, which allows detecting variables such as plectrum speed and acceleration. It is an advantage of embodiments of the present invention that using additional information, such as from a piezo sensor or from an accelerometer, results in a better prediction.

The processing unit may be used for analysis of data retrieved from the electrical sensor (e.g., for deriving which string was stroked, and/or whether the stroke of the string or strings is an up- or down-stroke). Additionally, it be used for analysis of data retrieved from the piezo sensor and/or from the accelerometer, if these are present in the plectrum.

The processing unit may be a CPU, a standard microchip, or it may be part of a controller, such as a microcontroller. Thus, the controller may receive data signals from the electrical detector, and it can be used for deriving whether the stroke on the string or strings is an up-stroke or a down-stroke, and/or for deriving which string was stroked.

The controller may be adapted for receiving data signals from a piezo sensor.

The controller may be adapted for receiving data signals from an accelerometer.

Alternatively, or in addition thereto, the controller also may be adapted for receiving data signals from a gyroscope, a hall effect sensor and/or an ultrasound or Doppler sensing system. In embodiments of the present system, the controller is configured for prediction of a stroke using reservoir computing.

It is an advantage of embodiments of the present invention that prediction of string onsets can be performed within a certain time frame, thus allowing manipulations to the audio signal, prior to the string actually sounding. It is an advantage of embodiments of the present invention that the manipulation does not only start after the string has started to sound but is prefacing the note onsets. It is an advantage of embodiments of the present invention that the system allows tapping into the micro-gestures of the playing instrumentalist and that parameters extracted can be used for musical expression. Additionally, such embodiments can be configured for performing data handling by recursive neural network.

A system according to some embodiments of the present invention furthermore comprises a module for further audio processing based on information obtained using the controller. In a further aspect of the present invention, the present invention provides a plectrum for stroking strings, the plectrum being provided with two separate electrodes, one electrode being positioned at one side of the plectrum body, the other electrode being position at the opposite side of the plectrum body.

The present invention provides use of a system according to embodiments of the fifth aspect of the present invention, and/or use of a plectrum according to a further aspect of the present invention, for sound manipulation. It is an advantage of embodiments of the present invention that the parameters measured using the system can be used for sound manipulations, thus enhancing the possibilities of playing a plucked string instrument, such as a guitar.

Some embodiments of the present invention provide the use of a system according to embodiments of the fifth aspect of the present invention, and/or use of a plectrum according to a further aspect of the present invention, for prediction of a stroke on the plucked string instrument, and/or for providing instructions within a teaching process for learning playing a plucked string instrument.

Particular and preferred aspects of the invention are set out in the accompanying independent and dependent claims. Features from the dependent claims may be combined with features of the independent claims and with features of other dependent claims as appropriate and not merely as explicitly set out in the claims.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

Brief description of the drawings

FIG. 1 illustrates a schematic representation the lateral side of a plectrum according to embodiments of the present invention.

FIG. 2 illustrates a system comprising a plectrum and an musical instrument according to embodiments of the present invention for data retrieval and analysis and/or audio manipulation.

FIG. 3 illustrates a system comprising a plectrum and an musical instrument according to embodiments of the present invention for data retrieval and analysis.

FIG. 4 illustrates the top side of a plectrum according to further embodiments of the present invention.

FIG. 5 illustrates the front, side, top and back of a plectrum according to embodiments of the present invention.

FIG. 6 illustrates a plectrum and a dedicated glove comprising connectors for the plectrum, according to embodiments of the present invention.

FIG. 7 illustrates examples of motion analysis allowed by the present invention.

The drawings are only schematic and are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for illustrative purposes.

Any reference signs in the claims shall not be construed as limiting the scope. In the different drawings, the same reference signs refer to the same or analogous elements.

Detailed description of illustrative embodiments

The present invention will be described with respect to particular embodiments and with reference to certain drawings but the invention is not limited thereto but only by the claims. The drawings described are only schematic and are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for illustrative purposes. The dimensions and the relative dimensions do not correspond to actual reductions to practice of the invention.

Furthermore, the terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequence, either temporally, spatially, in ranking or in any other manner. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other sequences than described or illustrated herein.

Moreover, the terms top, under and the like in the description and the claims are used for descriptive purposes and not necessarily for describing relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in other orientations than described or illustrated herein.

It is to be noticed that the term "comprising", used in the claims, should not be interpreted as being restricted to the means listed thereafter; it does not exclude other elements or steps. It is thus to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression "a device comprising means A and B" should not be limited to devices consisting only of components A and B. It means that with respect to the present invention, the only relevant components of the device are A and B.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

Similarly it should be appreciated that in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Where in embodiments of the present invention reference is made to a "pickup", reference is made to a sensor that senses vibrations of a metal string. These signals can be amplified and used to generate sound, and they can also be digitalized, stored and/or manipulated. This should not be confused with the "pick" which is a synonym for a "plectrum", in the frame of the present invention.

Where in embodiments of the present invention reference is made to a "up-stroke" and "down-stroke", reference is made to plucking techniques in which a plectrum plucks a string with an upward and downward movement of the plectrum, respectively.

In a first aspect, the present invention relates to a system for playing a musical instrument or assisting therein. The musical instrument may for example be a plucked string instrument, a string instrument, a wind instrument, a percussion instrument, a keyboard instrument, a lamellophone such as for example a kalimba, or a quintephone such as for example a theremin. The musical instrument may form part of the system or the system may be adapted for use with the musical instrument. According to embodiments of the present invention, the system comprises a detection system for detecting sensorimotor preparation for physical interaction of the player with the musical instrument. Sensorimotor preparation for physical interaction is defined as the movement of the instrumentalist or an item controlled by the instrumentalist for playing the musical instrument, prior to the actual physical interaction of the instrumentalist and the musical instrument or the item controlled by the instrumentalist and the musical instrument. The detection system may comprise a detection system for detecting a parameter representative for a relative movement or orientation, e.g. 1 dimensional, 2 dimensional or 3 dimensional movement, of the player or an additional aiding element with respect to the musical instrument. Movement of the player with respect to the musical instrument may for example comprise movement of a hand or foot with respect to the musical instrument, but alternatively or in addition thereto may also comprise movement of an aiding means such as for example a plectrum, a drumstick or a fiddle stick. Detectable preparatory movements may include: (changes in) distance between hand or aiding means from the sound-producing part of the musical instrument (e.g. a string, lamel, skin...), (changes in) inclination/orientation of the hand or aiding means, (changes in) speed of movement in any of the possible degrees of freedom. The detection system may comprise one or more of an optical camera or infrared camera, a radar, an ultrasone detector, a vibration sensor, a gyroscope, an accelerometer, a piezodetector or a hall sensor.

The detection system also comprises a controller for receiving data signals from the detection system and for predicting an interaction with the musical instrument, such as, but not limited to, playing a note, based on the detected sensorimotor preparation for interaction with the musical instrument, said predicting being performed before a physical interaction is performed with the musical instrument, using reservoir computing. Said processing using reservoir computing may be based on a recursive neural network. According to embodiments of the present invention, the processing takes as input information, e.g. data signals related to the sensorimotor preparation actions of the player for playing the musical instrument. It may be a distance to the musical instrument, a speed of approaching the musical instrument, the acceleration towards the musical instrument, the position with respect to the musical instrument or a specific part thereof, etc. The detection system may for example comprise an optical camera or infrared camera, a radar, an ultrasonic detector, a vibration sensor, a gyroscope, an accelerometer, a piezodetector or a hall sensor. It is an advantage of embodiments of the present invention that a prediction can be made, i.e. that the interaction with the musical instrument can be predicted and not only identified when the interaction takes place. Predicting of an interaction may comprise for example predicting any or a combination of an onset, a location of interaction with the musical instrument, an intensity of interaction with the musical instrument, an articulation variation or a timbral variation. The information regarding the prediction of the interaction may be used for audio processing, augmentation or providing training information to the player. In some embodiments, the system may be adapted for providing haptic feedback to the player of the musical instrument. The system may be adapted for providing such information using a further module or processor.

The system may comprise the features of a system as described in the further aspect, although embodiments are not limited thereto. In a further aspect, the present invention relates to system for playing a musical instrument or assisting therein, the system comprising a plucked string instrument with one or more conductive strings and a plectrum with an electrode on each of its front and back surfaces, the electrodes being electrically insulated from each other and separated by the thickness of the insulating plectrum body. The strings of the musical instrument and each electrode of the plectrum are connected to either a voltage source or to an electrical detector. The system can detect contact between one of the plectrum electrodes and one of the strings of the musical instrument. Further, it can detect which side of the plectrum has been used to hit the string. In some embodiments the system can discern which string is being plucked when each string is connected to a voltage source and put at different voltages, so that electrical contact between one of the electrodes of the plectrum and a particular string can be distinguished using the electrical detector. The present invention may present other configurations; for example an electrical detector may be connected to each string.

The system can further distinguish over different playing features such as up- and down-strokes, string selection and gesture velocity. This provides objective feedback and monitoring of playing techniques. The system may be used as a teaching tool, as self- monitoring learning tool, or as a means for sound manipulation as an augmented musical instrument, among other possibilities. The voltages of the string or plectrum electrodes typically is low enough not to have effect on the fingers of the player, when the player touches the strings with naked fingers or holds the plectrum. For example, typical voltages may for example range between +4V and - 4V, for example between 3 V and 0.1 V. The resistance of the human body is typically much higher than that of the electrodes and strings, which will be the conductive path in normal conditions.

In order to ensure no current losses through the player, some embodiments of the present invention include an extra isolating layer covering the plectrum electrodes. FIG. 1 shows the lateral view of a simple exemplary embodiment of an "augmented" plectrum 100 according to embodiments of the present invention. The electrodes 101, 102 of the plectrum may be copper electrodes, for example copper plates separated by an isolating piece or body 103 (e.g. the plectrum to which they are attached, deposited, or formed). An isolating covering layer 104 covers a portion of the electrodes on the plectrum gripping zone 105. The edge 106 of the plectrum used to strike the strings, hereinafter called the striking zone, comprises electrodes 101, 102 and should be uncovered. This ensures enough contact area between the electrodes and the strings. The isolating layer 104 may protect the connection between connections 107 (e.g. wires) and the electrodes 101, 102. The strings are typically conductive strings, such metal strings with low resistance (e.g. steel), further ensuring a low resistance path. The connections 107 normally are signal transmitting means, but they may be also connections to a voltage.

In embodiments of the present invention, at least one string is conductive. For example, only some of them may be conductive. For example the 4^th, 5^th and 6^th strings of a classic guitar may be conductive. In some embodiments, all strings are conductive. The conductive strings may comprise a metal or an alloy, such as steel and/or other materials such as copper, nickel, phosphor bronze, silver-plated bronze, etc., and they may be monofilament or wound strings (e.g. wound steel, nylon wound with copper, etc.).

In embodiments of the present invention, the strings are isolated from each other. There are a lot of ways to obtain this isolation, influenced by the type of musical instrument. For example, a Duesnberg tremolo tailpiece may comprise an electrically insulating bar (e.g. plastic, wooden, bone bar) for anchoring the strings. An intermediate insulating bar may push down the strings, keeping them in tension and in bridge saddles, which may also be insulated. A standard bridge saddle and/or tailpiece may comprise an insulating layer in contact with the string. Such insulating layer may be thin, reducing the influence on the vibration of the strings, and being easily removed and installed, allowing an easy and inexpensive adaptation of a standard musical instrument to the present invention. For example, it may be a polymeric layer, such as a thin plastic layer. In other embodiments, the tailpiece and/or bridge may comprise a plurality of saddles disconnected from each other, or it may comprise a single piece of wood, bone, polymer, mother-of-pearl, ceramic, bakelite, etc. Embodiments comprising other string fixation features, such as stoptail bridges or wrap-around tailpieces or the like, may have appropriate isolation means such as isolating layers or compositions. In other embodiments, the string wires themselves comprise an isolating layer (a nylon cover, textile, polymer, etc) on the first portion to be in contact with the bridges and tailpieces.

In some embodiments of the present invention, the strings are electrically connected to a voltage source and the electric signal is read through the electrodes of the plectrum. Each string may be connected to the same voltage, i.e. to a single voltage, in which case the electric detector of the plectrum can detect up- or down-strokes. In other embodiments, each string is connected to a different voltage (e.g. to a voltage divider, or to a voltage source each), so electrical contact between one string and one plectrum electrode can be detected, and the string be identified, using the electrical detector. For example, the strings may carry the voltages of 2.45 V, 2.11 V, 1.47 V, 0.95 V, 0.8 V and 0.32 V. Other values may be used. Strings can be electrically connected to conduction wires, for example by cylindrical screw clamps, or by clamps with a side insert allowing introduction and connection of the strings to the wires. Other suitable methods may include spring clips, etc. These embodiments allow an easy installation in any existing musical instrument, with no need for removing the strings from the guitar. In other embodiments, the strings and the connections may be soldered. In some other embodiments, the voltage source or sources (or the voltage source with a voltage divider) may also be integrated in the body of the musical instrument, for example in a battery case within the musical instrument, and the contact with the strings may be done through the clamps in the tailpiece. Thus, the tailpiece clamps provide string tension and fixation, as well as electric contact, in these embodiments.

The electrodes of the plectrum can be normally-open (NO) contacts. Touching the guitar strings is detected by identifying a voltage when an electrode is in contact with the string (thus closing a contact). By measuring the voltage when the plectrum touches a string (e.g. by connecting electric sensors to the electrodes of the plectrum or to each string), it can be determined which string is plucked. The separate electrodes on the front- and back-side of the pick allows distinguishing up-strokes from down-strokes in a particular string. The connection to electric sensors may be done through a standard input/output, for example a General Purpose Input/Output (GPIO). In embodiments of the present invention, this transmission may be fast through conductive paths formed by the electrodes and the strings. Low-resistance electrodes (e.g. copper) reduces latency time and speeds up signal transmission.

FIG. 2 shows an example of the system comprising a plectrum 100, a musical instrument 200 with four strings and an electrical detector system 210. Each of the strings are conductive and isolated from each other. Vibrations are read by the pickup 201. Each string 202, 203, 204, 205 is connected to a voltage source 206, 207, 208, 209, which may provide the same voltage to all strings or may provide four different voltages to each of them. The edge 106 of the plectrum for plucking strings is covered by two isolated electrodes, each of which is connected to the electrical detectors 211, 212 of the system 210. A couple of wires 107 can be used as signal transmitting means. These detectors may be an ohmmeter, current or voltage detector, potentiometer, a combination thereof, etc.

In other embodiments, as shown in FIG. 3, the electrodes of the plectrum 100 are connected to different voltages 110, 111. An exemplary set of values may be for example 3.3 V and 1.65 V in each electrode at each side of the plectrum. The plectrum may comprise connections for providing voltage, or it may comprise an integrated battery such a rechargeable battery. The contact between plectrum and string during plucking is then detected and read by connecting each of the strings to electric detectors 211, 212, 213, 214 of a detection system 210. The connection means 307 may preferably be a wired connection (by cylindrical screw clamps, spring clips, etc.), which reduces latency, for example via the ports of a GPIO. The signal transmission means 307 can be wireless (transducer, antenna, etc.). In case of wired connection, the connections and input/output ports, and even the detection system 210, may be integrated within the musical instrument.

The signals detected by the electrical detector system (voltage, current, etc.) comprise at least information regarding up- and down-stroke plucking. These signals, either obtained from the strings and/or plectrum, can be sent to a controller 220, such as a microcontroller, and/or to a processor, a CPU or any other processing means, which may be used to derive data regarding the action of playing the musical instrument, such as speed, up- and down-strokes, etc. It can be used to determine the particular string being played. In some embodiments, the controller 220 may have processing capabilities and/or may comprise a processor.

The present invention may also comprise means for signal conversion (ADC, DAC) and audio output control. For example, FIG. 3 shows an audio output module 230 connected directly to the musical instrument. The information from the detection system is digitized and stored, and can be used for analysis, teaching, etc. On the other hand, FIG. 2 shows an audio output 230 being connected to the musical instrument, and additionally comprising the controller 220. Data analyzed and processed by the controller may be used together with the (audio) signal from the musical instrument to add effects, improve signal processing, predict and anticipate notes, all with a very low latency. Thus, manipulation of audio signals may be done during performance.

A combination of such embodiments is also possible: for example, the controller 220 of FIG. 3 may also be used to manipulate the audio signal in the audio output 230, and the embodiment of FIG. 2 may be limited to data analysis and storage without manipulation of the audio.

The present invention is not limited to detection of contact between the plectrum electrodes and the string. In embodiments of the present invention, further sensors may be included. For example, one or more continuous Hall sensors may trace the movements of the plectrum, by installing a plurality of sensors near the stroking area, or in the plectrum itself for detecting movement with respect to e.g. the piezoelectric pickups of an electric musical instrument. In preferred embodiments, however, the plectrum comprises a piezo sensor. Additionally, it may comprise an accelerometer. Embodiments of the present invention may comprise means of transmitting all these additional signals (accelerometer signals, signals from the piezo sensor, etc.) to a processing unit or microcontroller. As before, the transmission may be wireless or wired.

Embodiments in which signal transmission is performed through wires, the response of the plectrum electrodes and the reading of the detector are fast, with little or no latency due to signal transmission. The present invention is not limited to wires, and other embodiments may comprise electrical bands. For example, electrical bands provided (e.g. printed, deposited, etc.) on a dedicated glove with electrical contacts on the surface of the glove finger which would hold the plectrum (e.g. on the thumb and index finger). If wires are used, they should be isolated between each other, for example by a simple isolating layer surrounding the wire. The wires may be concealed in a glove or held by a wristband or any other means, if it would be desirable to avoid accidental impact between the musical instrument strings and the wires.

Alternative embodiments may comprise signal transmission based on wireless communication, for example transmission based on radio-frequency transmission, Bluetooth, or any other system. For example, the plectrum may comprise a transmitter, and the signal may be read by a receiver. For example, an FID transceiver may be included in the plectrum and another transceiver in the musical instrument near the plucking area, which sends the signal to the processing unit. Alternatively or additionally, a Bluetooth transmitter may be included in the plectrum. The transmitter may be connected externally, e.g. via wires, to the plectrum, and send signal to a receiver. The receiver then feeds the signal or signals to the processing unit.

The processing unit (e.g. microcontroller) can be configured to track and store the signals, and may process them and analyze the style and method of playing. This can be useful for training and teaching purposes.

Further embodiments of the present invention may comprise analog to digital converters (ADCs), pickups, amplifiers, pre-amplifiers, etc. It may comprise, for example, one audio channel per string. An audio-to-GPIO board may provide audio signal onset per string. The audio signals can be converted, for example, to DC signals. Operational amplifier configurations and voltage converters can also be used. The processor may be a sound card, microcontroller, etc. It may comprise audio input/outputs, GPIO connections, connections to the electric sensors, to piezo sensors and to accelerometers. It may also comprise software and means for data analysis, treatment and audio output control.

Further embodiments may comprise using reservoir computing and data analysis and manipulation, as it will be seen in the section "applications".

In second aspect of the present invention, a plectrum comprising one electrode in each side is provided. The plectrum of the present invention comprise at least two electrodes 101, 102 on its edge, for example copper plates in each side of the plectrum body 103 as shown if FIG. 1. Thus, the non-conductive body 103 isolates the electrodes 101, 102 of each other. The present invention is not limited to two electrodes, and a third electrode may be placed for example in a side of the body 103 or in the edge of the wide part of the plectrum, for example for catching signals if the technique "pick slide" is used.

The body 103 may comprise plastic, wood, rubber nylon, polyamide-imide, methacrylate, ceramic etc. It may be a 1 mm, 2 mm or even 3 mm thick plectrum. Thick and/or stiff plectrums suffer less bending, which is advantageous for avoiding detachment or deformation of the electrodes. Plectrums typically have a gripping area 105 and an edge 106 used to pluck the strings. The electrodes 101, 102 must be present in both faces of the plectrum, on its edge 106, allowing contact between the electrodes and the string while plucking. The body may be further shaped so it has a side for optimal up-stroke and an opposite side for optimal down-stroke.

As shown in FIG. 2, other sensors 400, 410 may be included in the plectrum. For example, an accelerometer 400 may be included, preferably near the wide side of the plectrum, for example in the center. The accelerometer may detect movements in one, two, or three dimensions, for example left and right of the hand, distance from the strings and/or the height of string which is being plucked. A small piezo sensor 410, such as a piezoelectric disk, can also be included in the same position, for example in the gripping area. The piezoelectric disk must be chosen in order to obtain a good frequency-response, for example by choosing an appropriate diameter. Information acquired by the accelerometer and by the piezo sensor may comprise data regarding impact and other mechanical events, which can be also sent to the controller 220. These extra sensors can also be used for detecting up- and down-strokes, but the electrodes are more reliable and present less latency. Preferably, these sensors may be used to detect distance between the plectrum and the strings, movement speed, strength of impact, string height, etc.

Signal transmission means can also be included in the plectrum (either wireless or wired, as already explained), preferably in the wide side of the plectrum. These components and part of the electrodes may be covered and protected by an isolating shielding, such as a rubber cover, to avoid damage of the sensors and to avoid direct contact between the player and the electrodes and reducing or avoiding signal leaking. Of course, the shielding must leave a region of electrode to be able to reach the string. The plectrum may include means for indicating which side (which electrode) shall be used for up- and down-stroke, for example a written indication, a textured grip or a color code.

FIG. 4 shows an exemplary embodiment of an augmented plectrum 100 according to embodiments of the present invention, comprising electrodes (only one electrode 101 is shown) and an accelerometer 400 in the wide side of the plectrum, within the gripping area 105. The wire 107 is soldered to the electrode on the soldering area 401. This wire may be used to apply a voltage on the electrode 101, or to sense voltages from the strings and transmit the signal to an electrical detection system. Other wires 402 may transmit signals from the accelerometer directly to a controller. A connector 403 for the transmission means (or for providing voltage to the electrodes) may be included in the plectrum. FIG. 5 shows four different views of an exemplary plectrum according to some embodiments of the present invention. The front view 501 shows two electrodes 101, 102 separated and electrically isolated from each other. In the side view 502, it is shown a gripping zone 105, which may be isolated, and zone 106 for plucking the strings. The top view 503 shows the gripping area 105 and the plucking edge 106 comprising the top electrode 101. The electrode may cover only a small area at the edge, or it may cover almost half or half of the plectrum, or all the plectrum. At the rear, some embodiments of the present invention may comprise a connection area 403 for connecting for example wires, for signal transmission or for creating voltage in the electrodes. This portion may have a length of 9 mm, for example, or other values depending on the number of connections (e.g. for other included sensors) needed in the plectrum. The back view 504 shows the connection area 403. The shape of the connection 403 may be rectangular, for example with a long side of 9 mm, the shape of the plectrum may be drop-shaped in the top 503 and side 502 views, and be oval in the front 501 and back 504 views, but other shapes are possible, depending on the particular application. The width 511 of the plectrum may be 25 mm, and the thickness 512 may be 3.5 mm, which allow a good grip and handling. The length 513 may be 31 mm. The plucking edge area 106 may have a length 514 of 12 mm, and the electrodes 101, 102 may completely cover it, or only part of it (improving isolation). Alternatively, the electrodes may cover a wider area (if contact is allowed between the player's fingers and the electrodes).

In other alternative embodiments, the electrodes are not covered by an isolating layer in the gripping zone. FIG. 6 shows an augmented plectrum comprising two electrodes 101, 102 on the sides of a body 103, and neither a covering layer nor wires. In such embodiment, signal transmission from (or voltage to) the electrodes 101, 102 is provided by conductive contacts 601, 602 and bands 603, 604 provided on the surface of the fingers of a dedicated glove 600. The contacts and/or conductive bands may comprise conductive materials such as silver paste, metal plates, wires; these conductive materials may be printed, deposited, embroidered, or in general attached to the glove 600. In these cases, only the accelerometer or piezo sensor, if present, should be protected by a covering layer, but not the electrodes because direct contact between the electrodes and the glove contacts is desired. A system comprising such dedicated glove has the advantage that the wires do not disturb the natural plucking movements. Additionally, the electrode corresponding to the up-stoke movement always corresponds to the electrode in contact with the thumb, and vice-versa for the opposite electrode in contact with the index finger, avoiding the possibility of obtaining faulty up- and down-stroke signals due to holding the plectrum wrongly.

By way of illustration, embodiments of the present invention not being limited thereto, a number of examples will be described below. Whereas the examples may refer to particular applications and/or musical instruments, such as a guitar or drum, it will be understood that embodiments of the present invention are not limited thereto, but are for example also applicable to wind instruments, string instruments, keyboard instruments, lamellophones, or quintephones.

Exemplary application.

Examples of the first and further aspects include a computer program product, such as a processor as for example a microprocessor, a controller or alike, wherein the processor is adapted for receiving data signals from an electrical detector adapted for detecting the plucking of a string, the processor furthermore adapted for processing the data signals for deriving based thereon information regarding the playing of the musical instrument. Information regarding playing of the musical instrument may be information of the playing of the musical instrument in the past, e.g. has the correct string be plucked, but can also provide, as described in the first aspect and in further aspects, information of the playing of the musical instrument for the future, such as timing when following strokes are to be played, but also more complex analysis of the music played/to be played. The processor also may be adapted for receiving data signals for example of a piezo sensor and/or of an accelerometer.

The input signals from the electrodes, from the piezo sensor and/or accelerometer, and/or from the strings (either electrical signals, signals from the pickup or acoustic signals) can be processed, for example they can be combined and mixed with each other in a non-linear way using reservoir computing. This can be done in a processor or controller, or outside processors, for example using photonic devices or other means. For prediction of strokes or other types of complex information gathering for which complex processing of data signals is required, reservoir computing or recursive neural network approach can be used, but other methods such as linear methods also can be used.

String onsets can be predicted within a certain time frame, for example analyzing the speed and plucking trend. This can be used to retrieve information from micro-gestures of the player, allowing manipulations to the audio signal at the moment of producing the sound, as may be performed using a sound controlling system. In case of live performances, latency times should be very low. Common latency times may reach 5 to 10 milliseconds, which is still acceptable. The present invention offers these and even lower latency times thanks to fast computing and fast (e.g. wired) signal transmission.

This can be obtained by means of a module for audio processing added to the system, based on the information retrieved in the processing unit. The extracted parameters can be used to add effects on the music, such as amplification, tone and tune adjustment, etc.

However, embodiments of the present invention additionally allow audio manipulations prior to actually producing a sound in the string, prefacing the note onsets. Augmentation of the musical instrument can thus include anticipatory information. Sound processing can already be prepared before the string is going to be plucked, allowing augmented sound output in time with the human sense of expressive micro-timing. These embodiments predict the plucking action in order to provide sound processing in time, or even ahead, with this action. This is done by relying on the sensorimotor preparation for action, which in this case can be measured in the movement of the guitar plectrum (for example, when it moves towards one of the strings).

Data handling by recursive neural network can be configured in the processing unit.

In another aspect, use of the system according to the first or second aspect of the present invention is provided for sound manipulation, increasing the range of tools for musical expression.

In yet another example, the system is configured for providing audio processing or augmentation in a drum system. A camera system may detect, e.g. based on feature recognition of a top of a drumstick, the movement of one or more drumsticks with respect to a drum system and predict based thereon the moment that the drumstick will hit parts of the drum system. Other detection systems that can be used are piezo detectors or accelerometers for detecting a speed or acceleration of the drumsticks, hall sensors for detecting an approximity of the drumsticks to part of the drum system, etc. Based on the information calculated, processing of the corresponding sound may be performed. Such processing may comprise augmentation of the sound produced, using the information for providing feedback to the player, e.g. with respect to the fact that the sound was properly played, etc.

Example of implementation.

By way of illustration, embodiments of the present invention not being limited thereto, an example of an implementation of a system using reservoir computing is given below. Reservoir computing is a well known processing technique and the particulars of the example are only given for illustrative reasons and are not limiting. For the implementation shown, the example of a stroke prediction during guitar playing is used.

In order to combine the input sensors into making a prediction of a stroke before it occurs, we used the Reservoir Computing paradigm. In this paradigm, the input signals are first combined and mixed with each other in a non-linear way, in a near-chaotic system. This increases the richness of the input to a higher number of signals, in which it is easier to find a system which correlates to the desired output.

This can be done outside of processors, using for instance photonic devices or even a bucket of water. As long as the dynamical system used to mix up the input signals has the mathematical properties described in Lukosevicius and Jaeger, Computer Science Review 3 (2009) 127-149, the system is fit for pre-processing the input signals.

After this pre-processing of the input signals, a simple linear method can be used to discover even non-linear relationships between the input data. In the present example the more common recursive neural network (RNN) approach to reservoir computing was used. From the precent example, four input signals were received at a 3kHz sample rate to generate a prediction of an onset, namely the acceleration on 3 axis and a signal from the piezo element, whereby a setup using a piezo element and an accelerometer were used as described in the guitar setup described above. These input signals x^f at time t are then linearly projected with randomly chosen matrices l l hidden, l l/input and w/bias before being non-linearly transformed using a tanh activation function.

xpre ⁼ tanh(W hidden- ^xpre + ^ input- ^χ1 + ^wbias)

Consequently, these preprocessed signals are then classified using linear regression to approximate the desired output signal y^l , namely an estimation of the chance a stroke will happen in the next 50 ms.

= W. x_v ^l _re + w

In this equation, we need to optimize the parameters 1/1/ and w in order for y^t to have the desired behavior. Note that this estimation can and does take values outside of the interval [0, 1].

In our case, we needed to minimise the necessary calculations in both these formulas during evaluation time. To do this, following conditions were applied

We used a piecewise linear function to approximate the tanh activation function. As long as this piecewise linear function is output bounded and non-linear, it suffices for reservoir computing. The compiler used to program an Axoloti can expand this piecewise linear function into efficient machine code. This makes the binary code faster than a native tanh implementation.

We used sparse matrices l l hidden, l l/input and w/bias to minimise the number of multiplications needed to preprocess the input signals. As long as the mathematical properties of the dynamical system remain unchanged, this does not affect the performance of the reservoir.

In order for this system to work, we need to find the matrices 1/1/ and w which will do a linear transformation between the preprocessed input signals and the desired output signal. This is now easy to obtain using linear regression.

We recorded the input signals from the accelerometer and the piezo element, together with the onset detection. Using this onset detection, we created a ground truth matrix y of the desired output. This matrix has the value 1 when an onset will be detected within 50 ms, and a 0 otherwise. The input signals were normalised such that they have a mean of zero and a standard deviation of one, and a bias signal was added to obtain the input matrix x. The bias signal is necessary in order to reduce the search for the matrices 1/1/ and w to a single matrix l/l . Using these matrices, the matrix which minimises the mean squared error (MSE) between y and y on the training data can simply be found using the linear regression normal equation:

In this equation, there is a regularisation parameter Λ which determines the amount of generalisation done on the input data. We found that a value of 2 χ 10^"6 for A minimised the MSE on a validation set. Finally, in order to run this RNN on the Axoloti, we had to optimise the number of nodes in the hidden layer N, in our case the length of the matrix x_v ^l _re, and the sparsity of the matrices l/l/hidden, l/l input and w/bias. We found that the RNN using N = 50 and a sparsity of 0.04 minimised the MSE on the validation set, while still being able to run on the Axoloti.

This reservoir could be used to predict when an onset will happen in the next 50 ms, namely when y^l> 0.5. In order to establish the quality of the predictions, we looked at the distribution of the predictions vs. our ground truth, which is the real onset detection on the signals of the hexaphonic preamp.

To evaluate our setup, we recorded 2 min of playing. In this time, we recorded the occurrences of an onset detection on the preamp signal, together with the timings of the onset prediction using the reservoir, and the onset detection using only a threshold on the signal received from the piezo element. In total, there were about 300 onsets detected using this approach.

Using the reservoir computing paradigm for sensor fusion outperforms predictions using the piezo sensor. To dig further into this, we compared the precision and recall of both methods and look whether the timing of these methods is accurate. Therefore, we defined predictions which came more than 100 ms before an actual onset as a false positive, predictions between 100 and 75 ms as too soon, predictions between 75 and 25 ms as correct and predictions between 25 and 0ms as too late. Vice versa, we look at the number of actual onsets, which were preceded by a correct prediction (both too soon, too late and correct) and compare this to the number of onsets which were not preceded by a prediction.

Using sensor fusion with reservoir computing improves the results considerably. A first observation is that the reservoir produces more timely predictions. Using only the piezo element predictions are more often too late.

A second observation is that both the piezo method and the reservoir have a good recall (of about 95%), but that the precision the reservoir achieves is higher. We reckon that the overall low precision is also related to the accuracy of the onset detection, where plucks on the guitar did not provide enough energy to be detected by the onset detection.

Examples of use.

The present invention may be applied to any plucked string instrument, for example a veena, banjo, ukulele, guitar, harp, lute, mandolin, sitar, etc.

When plucking a string, four phases take place: (a) approach of the plectrum, (b) impact, (c) release, and (d) vibration onset. If an accelerometer is included, the first phase (a) may already be detected and processed by the system. The second phase (b) takes place on the first contact between the plectrum and the string, and the first timed event can be detected as the circuit closes due to the electric contact between one of the electrodes and the string. The system provides discrimination between up- or down-stroke before any sound is even produced. If a piezo sensor is included, it may also detect the impact and the controller may derive information relating to the strength of the impact, for example whether it was hitting a muted string, or the player was executing a pinch harmonic, etc. The third phase (c) produces another onset in the piezo detector and a voltage of zero (open circuit). Finally, in the fourth phase (d), the string starts vibrating, producing the onset detection of the audio signal. The accelerometer can also detect especial executions such as "pick slide". Data from the piezo sensor and/or the accelerometer may provide a continuous signal, as opposed to the discrete signal of the plectrum electrodes. These continuous signals can be used to generate training data for the reservoir, increasing prediction accuracy.

FIG. 7 shows an example of musical devices and movements by the hand 700 of a player on a musical instrument 200 (a guitar) which can be detected by the present invention. Strokes can be detected before sound is produced, if a piezo sensor is included in the plectrum. Upstrokes 701 and down-strokes 702 can always be detected and distinguished by the electrodes of the plectrum before a note has played. This information may be included in the data reservoir, and it can be used to start sound manipulation at the same time, or even before, the acoustic signal is produced. An included accelerometer can track the movements in the X, Y and Z direction, as indicated in FIG. 4, which corresponds to the position of the plectrum in the directions right-left (direction X), the distance 703 towards the strings (direction Y) and the up- down movement, indicating the height of the string (direction Z).

In a first simple example of use, the detection of up- and down-stroke can be used by the controller to produce variations in the sound. For example, the difference in sound between up- and down-strokes in a guitar is, although subtle, an important feature of the distinct guitar sound. In most existing sound manipulations for guitar, this aspect is not taken into account. Sound manipulation generally does not change depending on whether a note was produced by an up- or down-stroke. In embodiments of the present invention, up- and down- stroke detection can be implemented, and it can be used to open and close a low-pass filter.

In an example of a use of a system comprising a 2-pole resonant low pass filter, the cut-off frequency of the filter can be controlled by the probability level of the reservoir's stroke prediction. This probability level indicates how high the probability is that a string will be struck in the next 50 milliseconds. The resulting sound is similar to that of a so-called "auto-wah" effect, but it differs in how it reacts to the playing. This effect has a more subtle relation to the playing gestures. While a traditional auto-wah effect simply responds to the amplitude envelope of played notes, the present invention provides a direct response to the gesture of the guitar plectrum. It provides a more flexible way of interacting with the low-pass filter. The cut-off frequency can be controlled after a note has been played, but it is also congruous with played note onsets.

In a further example, notes can be anticipated in sound before they are played. Thus, the system and/or plectrum of the present invention may be used for prediction of a stroke on the plucked string instrument. For example, through the combined use of predicted and detected onsets, sounds are introduced before played notes. The predicted onset starts a sound that continues until the detected onset stops it. This way, onsets of notes can be manipulated without disturbing the actual sounding notes. The integrity of the timing of the played notes is preserved.

In a further example, the prediction may be used in portamento (glide), allowing the pitch slope between a first note and a target note to start before the target note is actually played.

Certain embodiments of the present invention can be used for polyphonic playing and strumming, allowing prediction of pick-to-string collisions several steps beforehand. For example, a musical instrument comprising conductive frets, when playing a chord the strings can be in electrical contact. If two different voltages are applied to the plectrum electrodes, and the signal is being read directly on the strings, the system can simultaneously identify all the strings being pressed and played in a chord.

The gesture characteristics can be separated in ancillary gestures and the opposing sound producing gestures. While in most applications, sound manipulation is performed by addition of extraneous gestures, the present system provides an augmented musical instrument and allows music manipulation with no need of movements extraneous to the act of playing. For example, there is no need of stepping on pedals, applying pressure on the plectrum, activation of buttons or knobs or other gestures, allowing the player to better focus on musical expression and increasing effectivity.

Examples of use for teaching.

In a further use of a system and/or plectrum of the present invention, instructions within a teaching process can be provided. Embodiments of the present invention can enhance the learning of the standard way of playing the musical instrument. The use of sensors introduces the possibility to quantitatively monitor different aspects (e.g. technical, expressive) of playing. As such, they can provide objective feedback that bypasses perceptual and interpretative difficulties that are often involved in interpersonal feedback processes, such as the conventional teacher feedback. Next to stimulating the learner's self-monitoring abilities, this kind of feedback may contribute to self-regulation skills and to the learner's autonomy. The present invention can provide feedback on the use of up-stroke vs. down-stroke plucking. Developing awareness of different uses of plucking is an essential part of the learning process. For example, it can be used for training of "alternate picking", which relies in the strict alternation of upstrokes and downstrokes, which can be monitored thanks to the present invention. Another use of the present invention comprises enlarging the sonic difference between up and down-strokes, so that the learner automatically becomes more aware of plucking patterns. For example, visual cues in the form of colored stage lighting that changes according to up-or down-strokes can be implemented, (e.g. red light for up-stroke, blue light for down-stroke). Such an approach may stimulate a more learner-centered approach in which the learner can explore and experiment with the various effects of plucking. In this way, the use of the present invention may complement more traditional approaches in which plucking may be taught, and by instructing the learner to execute strokes according to this model. Furthermore the note prefixes and the non-plucking gesture control can be used to increase awareness of gestural aspects of playing, and to stimulate the deliberate use of expressive and communicative gestures through sonic rewards. In addition, this may increase awareness about the efficiency of the plucking gestures of a player, revealing aspects of used energy and distance traveled by the plectrum. In this way, the present invention contributes to developing other techniques such as "economy picking" and "sweep picking", as well as detecting dangerous habits such as wrist locking when plucking the strings. Thus, the present invention may reduce risk of injury in students.

Claims

A system for playing a musical instrument, the system comprising

a detection system for detecting sensorimotor preparation for physical interaction of a player with the musical instrument,

a controller for receiving data signals representative for the sensorimotor preparation for physical interaction from the detection system and for predicting a physical interaction with the musical instrument prior to said physical interaction with the musical instrument based on the received data signals, said predicting being performed using reservoir computing.

A system according to claim 1, wherein said predicting is performed using a recursive neural network.

A system according to any of the previous claims, wherein said predicting a physical interaction comprises predicting any or a combination of an onset, a location of physical interaction with the musical instrument, an intensity of physical interaction with the musical instrument, an articulation variation or a timbral variation, pitch choices in melodic or harmonic playing and/or their intervallic relationship to other pitch choices, rhythmic note values in relation to the meter and tempo of the current musical context, data collection for statistical analysis and community interactions or educational purposes.

A system according to any of the previous claims, wherein the system comprises furthermore a module for further audio processing based on the predicted physical interaction obtained using the controller.

A system according to any of the previous claims, wherein the controller furthermore is configured for inducing an augmentation of the sound produced by the musical instrument based on the predicted physical interaction with the musical instrument. A system according to any of the previous claims, wherein the controller furthermore is configured for providing feedback to the player for training purposes, direction purposes or conducting purposes.

A system according to any of the previous claims, wherein the detection system for detecting sensorimotor preparation for physical interaction with the musical instrument is a detection system for detecting movement of the player with respect to the musical instrument.

8. A system according to any of the previous claims, wherein the detection system comprises one or more of an optical camera or infrared camera, a radar, an ultrasone detector, a vibration sensor, a gyroscope, an accelerometer, a piezodetector or a hall sensor.

9. A system according to any of the previous claims, wherein the system furthermore is adapted for providing haptic feedback to the player.

10. A system according to any of the previous claims, wherein the musical instrument is any of a string based plucking instrument, a string instrument, a percussion instrument or a wind instrument.

11. A system according to any of the previous claims, the system comprising

- a plectrum (100) for stroking strings, the plectrum being provided with two electrically separated electrodes (101, 102) and a plectrum body (103), one electrode being positioned at one side of the plectrum body (103), the other electrode being positioned at an opposite side of the plectrum body (103), and

- a plucked string instrument (200) wherein at least 2 of the strings (202, 203, 204, 205) are made of electrically conductive material and wherein the at least 2 of the strings are electrically insulated from each other,

wherein either the electrodes of the plectrum (100) or the at least 2 of the strings (202, 203,

204, 205) of the plucked string instrument are electrically connected to a voltage source

(110, 111, 206, 207, 208, 209), and wherein the remaining of the electrodes of the plectrum (100) or the at least 2 of the strings (202, 203, 204, 205) are connected to an electrical detector (210) for detecting an electrical contact between one of the two electrodes (101, 102) and one of the at least 2 of the strings (202, 203, 204, 205), wherein furthermore the detection system for detecting sensorimotor preparation for physical interaction of a player with the musical instrument is integrated in the plectrum.

12. A system according to claim 11, wherein the detection system furthermore comprises a piezo sensor (400) and/or an accelerometer (410).

13. A system according to any of claims 11 to 12, wherein the controller is adapted for predicting whether a stroke on the string or strings will be an up-stroke or a down-stroke, and/or for deriving which string will be stroke.

14. A system according to any of the previous claims, wherein the controller is adapted for detecting ancillary gestures of the player indicative for control of a sound controlling system.

15. Use of a system according to any of claims 1 to 14 for sound manipulation.

16. Use of a system according to any of claims 1 to 14 for training purposes.

17. Use of a system according to any of claims 11 to 14 for prediction of a stroke on a plucked string instrument.