This is ipt~ project page. ipt~ is part of the ERC REACH project.

ipt~ is a tool that leverages deep learning models for automatic classification, trained on a vast dataset of instrumental sounds. It already includes a flute and an electric guitar model. The flute model, the most advanced and best-performing to date, generalizes effectively and effortlessly to wind instruments. The electric guitar model enables the recognition of a wide range of contemporary techniques, including those played with accessories. ipt~ can deliver responses in record time (from a few milliseconds to 100-200ms) with impressive success rates (> 95%). As such, it remains a unique tool in the world, likely to be very popular with artists, designers, and researchers in the field of intelligent interaction with AI and mixed music.

It marks an important milestone toward the development of listening systems capable of understanding high-level instrumental performance by mapping instrumental morphologies over time, and, more broadly, spectral morphologies in general. ipt~ thus enables co-creative interactive systems to interact more coherently and with a better understanding of the musical context. The resulting recognition stream from ipt~ can drive synthesis, trigger events in score-following systems such as Antescofo, or feed generative agents like Somax2, opening a continuous dialogue between the performer's sonic gesture and the machine's response.

ipt~ integrated into a general workflow that includes Somax2 and other tools:

• Install last release from Github : download last release
• Train your own model using ipt_recognition repository

• Introduction to IPT~
• Demonstration
• Somax2 and IPT~
• Two more videos are coming..! "How to train your own IPT model" and "Advanced usage with Somax2 and Antescofo".

Open-source datasets used to train ipt~:

• GFD, extensive flute IPT dataset. Performers: Kanami Koga & Nicolas Brochec. Sound Engineer: Will Howie.
• EG-IPT, extensive e-guitar IPT dataset. Performer: Marco Fiorini. Sound Engineer: Riccardo Pasini.

• More refined interfaces, APIs, and usage templates; completion of the tutorials collection
• Expansion of available instrument models
• Development and distribution of the pipo ipt module with Diemo Schwarz
• Development and distribution of the ipt VAMP Plugin with Pierre Guillot
• Direct use on an unknown instrument without training using zero-shot or few-shot approaches, leveraging CLAP (Contrastive Language-Audio Pretraining) technology

• Nicolas Brochec Advancing Human-Computer Interaction in Mixed Music: A Deep Learning Approach to Real-Time Instrumental Playing Technique Recognition, Musicology and performing arts. Tokyo University of the Arts, 2026. English.
• Nicolas Brochec, Jean-Louis Giavitto. Automatic Hybrid Following in Real-Time Mixed Music: A Case Study with Antescofo and ipt˜ for Flute Playing Techniques. International Computer Music Conference (ICMC 2026), May 2026, Hamburg, Germany.
• Marco Fiorini, Nicolas Brochec, Joakim Borg, Riccardo Pasini. Introducing EG-IPT and ipt~: a novel electric guitar dataset and a new Max/MSP object for real-time classification of instrumental playing techniques. New Interfaces for Musical Expression (NIME 2025), Jun 2025, Canberra, Australia.
• Nicolas Brochec, Marco Fiorini, Mikhail Malt, Gérard Assayag. Interactive Music Co-Creation with an Instrumental Technique-Aware System: A Case Study with Flute and Somax2. International Computer Music Conference (ICMC 2025), Jun 2025, Boston (MA), United States.
• Marco Fiorini, Nicolas Brochec. Guiding Co-Creative Musical Agents through Real-Time Flute Instrumental Playing Technique Recognition. Sound and Music Computing Conference (SMC 2024), Jul 2024, Porto, Portugal.
• Nicolas Brochec, Tsubasa Tanaka, Will Howie. Microphone-based Data Augmentation for Automatic Recognition of Instrumental Playing Techniques. International Computer Music Conference (ICMC 2024), Jul 2024, Seoul, South Korea.
• Nicolas Brochec, Tsubasa Tanaka. Toward Real-Time Recognition of Instrumental Playing Techniques for Mixed Music: A Preliminary Analysis. International Computer Music Conference (ICMC 2023), Oct 2023, Shenzhen, China.

• Somax2 and ipt~ demo at SMC 2024
• ipt~ demo at NIME 2025
• C API of ipt~

ipt~ is born between Tokyo University of the Arts and IRCAM Music Representations Team. This project is supported by the ERC REACH (Raising Co-creativity in Cyber-Human Musicianship), directed by Gérard Assayag.

ipt~ created by Nicolas Brochec (Project Leader, R&D), Marco Fiorini (R&D), Joakim Borg (R&D).

Thanks to Kanami Koga and Simone Conforti for their continuous expertise on the flute.

Thanks to Nicolas Souchal for its continuous expertise on the trumpet.

Thanks to George Lewis, Steve Lehman, and Joëlle Léandre for their exploratory usage of ipt~ in concerts.