Essentia models¶
This page provides a list of pre-trained models available in Essentia for various music and audio analysis tasks. To use Essentia with TensorFlow support refer to the guide on Using machine learning models.
Click on the models below to access the weights (.pb) and metadata (.json) files, as well as example code snippets.
Additional legacy models are available in our model repository.
Some models are also available in TensorFlow.js (tfjs.zip) and ONNX (.onnx) formats.
As this is an ongoing project, we expect to keep adding new models and improved versions of the existing ones. These changes are tracked in this CHANGELOG.
All the models created by the MTG are licensed under CC BY-NC-SA 4.0 and are also available under proprietary license upon request. Check the LICENSE of the models.
Follow this link to see interactive demos of some of the models. Some of our models can work in real-time, opening many possibilities for audio developers. For example, see Python examples for MusiCNN-based music auto-tagging and classification of a live audio stream.
If you use any of the models in your research, please cite the following paper:
@inproceedings{alonso2020tensorflow,
title={Tensorflow Audio Models in {Essentia}},
author={Alonso-Jim{\'e}nez, Pablo and Bogdanov, Dmitry and Pons, Jordi and Serra, Xavier},
booktitle={International Conference on Acoustics, Speech and Signal Processing ({ICASSP})},
year={2020}
}
Feature extractors¶
AudioSet-VGGish¶
Audio embedding model accompanying the AudioSet dataset, trained in a supervised manner using tag information for YouTube videos.
Models:
⬇️ audioset-vggish
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictVGGish audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = model(audio)
Discogs-EffNet¶
Audio embedding models trained with classification and contrastive learning objectives using an in-house dataset annotated with Discogs metadata.
The classification model was trained to predict music style labels.
The contrastive learning models were trained to learn music similarity capable of grouping audio tracks coming from the same artist, label (record label), release (album), or segments of the same track itself (self-supervised learning).
Additionally, multi was trained in multiple similarity targets simultaneously.
Models:
⬇️ discogs-effnet-bs64
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = model(audio)
Note: We provide models operating with a fixed batch size of 64 samples since it was not possible to port the version with dynamic batch size from ONNX to TensorFlow. Additionally, an ONNX version of the model with dynamic batch size is provided.
MAEST¶
Music Audio Efficient Spectrogram Transformer (MAEST) trained to predict music style labels using an in-house dataset annotated with Discogs metadata.
We offer versions of MAEST trained with sequence lengths ranging from 5 to 30 seconds (5s, 10s, 20s, and 30s), and trained starting from different intial weights: from random initialization (fs), from DeiT pre-trained weights (dw), and from PaSST pre-trained weights (pw). Additionally, we offer a version of MAEST trained following a teacher student setup (ts).
According to our study discogs-maest-30s-pw, achieved the most competitive performance in most downstream tasks (refer to the paper for details).
Models:
⬇️ discogs-maest-30s-pw
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-30s-pw-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-30s-pw-ts
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-30s-pw-ts-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-20s-pw
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-20s-pw-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-10s-pw
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-10s-pw-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-10s-fs
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-10s-fs-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-10s-dw
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-10s-dw-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)⬇️ discogs-maest-5s-pw
Model trained with a multi-label classification objective targeting 400 Discogs styles.
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMAEST audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMAEST(graphFilename="discogs-maest-5s-pw-1.pb", output="StatefulPartitionedCall:7") embeddings = model(audio)
Note: It is possible to retrieve the output of each attention layer by setting output=StatefulParitionedCall:n , where n is the index of the layer (starting from 1).
The output from the attention layers should be interpreted as [batch_index, 1, token_number, embeddings_size]
, where the first and second tokens (i.e., [0, 0, :2, :] ) correspond to the CLS and DIST tokens respectively, and the following ones to input signal.
OpenL3¶
Audio embedding models trained on audio-visual correspondence in a self-supervised manner.
There are different versions of OpenL3 trained on environmental sound (env) or music (music) datasets, using 128 (mel128) or 256 (mel256) mel-bands, and with 512 (emb512) or 6144 (emb6144) embedding dimensions.
Models:
MSD-MusiCNN¶
A Music embedding extractor based on auto-tagging with the 50 most common tags of the Million Song Dataset.
Models:
⬇️ msd-musicnn
Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = model(audio)
Classifiers¶
Classification and regression models based on embeddings.
Instead of working with mel-spectrograms, these models require embeddings as input.
The name of these models is a combination of the classification/regression task and the name of the embedding model that should be used to extract embeddings (<classification_task>-<embedding_model>).
Note: TensorflowPredict2D has to be configured with the correct output layer name for each classifier. Check the attached JSON file to find the name of the output layer on each case.
Music genre and style¶
Genre Discogs400¶
Music style classification by 400 styles from the Discogs taxonomy:
Blues: Boogie Woogie, Chicago Blues, Country Blues, Delta Blues, Electric Blues, Harmonica Blues, Jump Blues, Louisiana Blues, Modern Electric Blues, Piano Blues, Rhythm & Blues, Texas Blues
Brass & Military: Brass Band, Marches, Military
Children's: Educational, Nursery Rhymes, Story
Classical: Baroque, Choral, Classical, Contemporary, Impressionist, Medieval, Modern, Neo-Classical, Neo-Romantic, Opera, Post-Modern, Renaissance, Romantic
Electronic: Abstract, Acid, Acid House, Acid Jazz, Ambient, Bassline, Beatdown, Berlin-School, Big Beat, Bleep, Breakbeat, Breakcore, Breaks, Broken Beat, Chillwave, Chiptune, Dance-pop, Dark Ambient, Darkwave, Deep House, Deep Techno, Disco, Disco Polo, Donk, Downtempo, Drone, Drum n Bass, Dub, Dub Techno, Dubstep, Dungeon Synth, EBM, Electro, Electro House, Electroclash, Euro House, Euro-Disco, Eurobeat, Eurodance, Experimental, Freestyle, Future Jazz, Gabber, Garage House, Ghetto, Ghetto House, Glitch, Goa Trance, Grime, Halftime, Hands Up, Happy Hardcore, Hard House, Hard Techno, Hard Trance, Hardcore, Hardstyle, Hi NRG, Hip Hop, Hip-House, House, IDM, Illbient, Industrial, Italo House, Italo-Disco, Italodance, Jazzdance, Juke, Jumpstyle, Jungle, Latin, Leftfield, Makina, Minimal, Minimal Techno, Modern Classical, Musique Concrète, Neofolk, New Age, New Beat, New Wave, Noise, Nu-Disco, Power Electronics, Progressive Breaks, Progressive House, Progressive Trance, Psy-Trance, Rhythmic Noise, Schranz, Sound Collage, Speed Garage, Speedcore, Synth-pop, Synthwave, Tech House, Tech Trance, Techno, Trance, Tribal, Tribal House, Trip Hop, Tropical House, UK Garage, Vaporwave
Folk, World, & Country: African, Bluegrass, Cajun, Canzone Napoletana, Catalan Music, Celtic, Country, Fado, Flamenco, Folk, Gospel, Highlife, Hillbilly, Hindustani, Honky Tonk, Indian Classical, Laïkó, Nordic, Pacific, Polka, Raï, Romani, Soukous, Séga, Volksmusik, Zouk, Éntekhno
Funk / Soul: Afrobeat, Boogie, Contemporary R&B, Disco, Free Funk, Funk, Gospel, Neo Soul, New Jack Swing, P.Funk, Psychedelic, Rhythm & Blues, Soul, Swingbeat, UK Street Soul
Hip Hop: Bass Music, Boom Bap, Bounce, Britcore, Cloud Rap, Conscious, Crunk, Cut-up/DJ, DJ Battle Tool, Electro, G-Funk, Gangsta, Grime, Hardcore Hip-Hop, Horrorcore, Instrumental, Jazzy Hip-Hop, Miami Bass, Pop Rap, Ragga HipHop, RnB/Swing, Screw, Thug Rap, Trap, Trip Hop, Turntablism
Jazz: Afro-Cuban Jazz, Afrobeat, Avant-garde Jazz, Big Band, Bop, Bossa Nova, Contemporary Jazz, Cool Jazz, Dixieland, Easy Listening, Free Improvisation, Free Jazz, Fusion, Gypsy Jazz, Hard Bop, Jazz-Funk, Jazz-Rock, Latin Jazz, Modal, Post Bop, Ragtime, Smooth Jazz, Soul-Jazz, Space-Age, Swing
Latin: Afro-Cuban, Baião, Batucada, Beguine, Bolero, Boogaloo, Bossanova, Cha-Cha, Charanga, Compas, Cubano, Cumbia, Descarga, Forró, Guaguancó, Guajira, Guaracha, MPB, Mambo, Mariachi, Merengue, Norteño, Nueva Cancion, Pachanga, Porro, Ranchera, Reggaeton, Rumba, Salsa, Samba, Son, Son Montuno, Tango, Tejano, Vallenato
Non-Music: Audiobook, Comedy, Dialogue, Education, Field Recording, Interview, Monolog, Poetry, Political, Promotional, Radioplay, Religious, Spoken Word
Pop: Ballad, Bollywood, Bubblegum, Chanson, City Pop, Europop, Indie Pop, J-pop, K-pop, Kayōkyoku, Light Music, Music Hall, Novelty, Parody, Schlager, Vocal
Reggae: Calypso, Dancehall, Dub, Lovers Rock, Ragga, Reggae, Reggae-Pop, Rocksteady, Roots Reggae, Ska, Soca
Rock: AOR, Acid Rock, Acoustic, Alternative Rock, Arena Rock, Art Rock, Atmospheric Black Metal, Avantgarde, Beat, Black Metal, Blues Rock, Brit Pop, Classic Rock, Coldwave, Country Rock, Crust, Death Metal, Deathcore, Deathrock, Depressive Black Metal, Doo Wop, Doom Metal, Dream Pop, Emo, Ethereal, Experimental, Folk Metal, Folk Rock, Funeral Doom Metal, Funk Metal, Garage Rock, Glam, Goregrind, Goth Rock, Gothic Metal, Grindcore, Grunge, Hard Rock, Hardcore, Heavy Metal, Indie Rock, Industrial, Krautrock, Lo-Fi, Lounge, Math Rock, Melodic Death Metal, Melodic Hardcore, Metalcore, Mod, Neofolk, New Wave, No Wave, Noise, Noisecore, Nu Metal, Oi, Parody, Pop Punk, Pop Rock, Pornogrind, Post Rock, Post-Hardcore, Post-Metal, Post-Punk, Power Metal, Power Pop, Power Violence, Prog Rock, Progressive Metal, Psychedelic Rock, Psychobilly, Pub Rock, Punk, Rock & Roll, Rockabilly, Shoegaze, Ska, Sludge Metal, Soft Rock, Southern Rock, Space Rock, Speed Metal, Stoner Rock, Surf, Symphonic Rock, Technical Death Metal, Thrash, Twist, Viking Metal, Yé-Yé
Stage & Screen: Musical, Score, Soundtrack, Theme
Models:
⬇️ genre_discogs400
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="genre_discogs400-discogs-effnet-1.pb", input="serving_default_model_Placeholder", output="PartitionedCall:0") predictions = model(embeddings)
MTG-Jamendo genre¶
Multi-label classification with the genre subset of MTG-Jamendo Dataset (87 classes):
60s, 70s, 80s, 90s, acidjazz, alternative, alternativerock, ambient, atmospheric, blues, bluesrock, bossanova, breakbeat,
celtic, chanson, chillout, choir, classical, classicrock, club, contemporary, country, dance, darkambient, darkwave,
deephouse, disco, downtempo, drumnbass, dub, dubstep, easylistening, edm, electronic, electronica, electropop, ethno,
eurodance, experimental, folk, funk, fusion, groove, grunge, hard, hardrock, hiphop, house, idm, improvisation, indie,
industrial, instrumentalpop, instrumentalrock, jazz, jazzfusion, latin, lounge, medieval, metal, minimal, newage, newwave,
orchestral, pop, popfolk, poprock, postrock, progressive, psychedelic, punkrock, rap, reggae, rnb, rock, rocknroll,
singersongwriter, soul, soundtrack, swing, symphonic, synthpop, techno, trance, triphop, world, worldfusion
Models:
⬇️ mtg_jamendo_genre-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mtg_jamendo_genre-discogs-effnet-1.pb") predictions = model(embeddings)
Moods and context¶
Approachability¶
Music approachability predicts whether the music is likely to be accessible to the general public (e.g., belonging to common mainstream music genres vs. niche and experimental genres).
The models output rather two (approachability_2c) or three (approachability_3c) levels of approachability or continous values (approachability_regression).
Models:
⬇️ approachability_2c-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="approachability_2c-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ approachability_3c-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="approachability_3c-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ approachability_regression-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="approachability_regression-discogs-effnet-1.pb", output="model/Identity") predictions = model(embeddings)
Engagement¶
Music engagement predicts whether the music evokes active attention of the listener (high-engagement “lean forward” active listening vs. low-engagement “lean back” background listening).
The models output rather two (engagement_2c) or three (engagement_3c) levels of engagement or continuous (engagement_regression) values (regression).
Models:
⬇️ engagement_2c-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="engagement_2c-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ engagement_3c-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="engagement_3c-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ engagement_regression-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="engagement_regression-discogs-effnet-1.pb", output="model/Identity") predictions = model(embeddings)
Arousal/valence DEAM¶
Music arousal and valence regression with the DEAM dataset (2 dimensions, range [1, 9]):
valence, arousal
Models:
⬇️ deam-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="deam-msd-musicnn-2.pb", output="model/Identity") predictions = model(embeddings)⬇️ deam-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="deam-audioset-vggish-2.pb", output="model/Identity") predictions = model(embeddings)
Arousal/valence emoMusic¶
Music arousal and valence regression with the emoMusic dataset (2 dimensions, range [1, 9]):
valence, arousal
Models:
⬇️ emomusic-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="emomusic-msd-musicnn-2.pb", output="model/Identity") predictions = model(embeddings)⬇️ emomusic-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="emomusic-audioset-vggish-2.pb", output="model/Identity") predictions = model(embeddings)
Arousal/valence MuSe¶
Music arousal and valence regression with the MuSE dataset (2 dimensions, range [1, 9]):
valence, arousal
Models:
⬇️ muse-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="muse-msd-musicnn-2.pb", output="model/Identity") predictions = model(embeddings)⬇️ muse-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="muse-audioset-vggish-2.pb", output="model/Identity") predictions = model(embeddings)
Danceability¶
Music danceability (2 classes):
danceable, not_danceable
Models:
⬇️ danceability-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="danceability-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ danceability-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="danceability-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ danceability-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="danceability-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ danceability-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="danceability-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Aggressive¶
Music classification by mood (2 classes):
aggressive, non_aggressive
Models:
⬇️ mood_aggressive-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_aggressive-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_aggressive-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_aggressive-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_aggressive-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_aggressive-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_aggressive-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_aggressive-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Happy¶
Music classification by mood (2 classes):
happy, non_happy
Models:
⬇️ mood_happy-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_happy-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_happy-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_happy-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_happy-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_happy-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_happy-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_happy-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Party¶
Music classification by mood (2 classes):
party, non_party
Models:
⬇️ mood_party-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_party-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_party-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_party-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_party-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_party-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_party-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_party-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Relaxed¶
Music classification by mood (2 classes):
relaxed, non_relaxed
Models:
⬇️ mood_relaxed-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_relaxed-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_relaxed-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_relaxed-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_relaxed-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_relaxed-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_relaxed-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_relaxed-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Sad¶
Music classification by mood (2 classes):
sad, non_sad
Models:
⬇️ mood_sad-audioset-yvggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_sad-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_sad-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_sad-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_sad-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_sad-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_sad-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_sad-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Moods MIREX¶
Music classification by mood with the MIREX Audio Mood Classification Dataset (5 mood clusters):
1. passionate, rousing, confident, boisterous, rowdy
2. rollicking, cheerful, fun, sweet, amiable/good natured
3. literate, poignant, wistful, bittersweet, autumnal, brooding
4. humorous, silly, campy, quirky, whimsical, witty, wry
5. aggressive, fiery, tense/anxious, intense, volatile, visceral
Models:
⬇️ moods_mirex-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="moods_mirex-msd-musicnn-1.pb", input="serving_default_model_Placeholder", output="PartitionedCall") predictions = model(embeddings)⬇️ moods_mirex-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="moods_mirex-audioset-vggish-1.pb", input="serving_default_model_Placeholder", output="PartitionedCall") predictions = model(embeddings)
MTG-Jamendo mood and theme¶
Multi-label classification with mood and theme subset of the MTG-Jamendo Dataset (56 classes):
action, adventure, advertising, background, ballad, calm, children, christmas, commercial, cool, corporate, dark, deep,
documentary, drama, dramatic, dream, emotional, energetic, epic, fast, film, fun, funny, game, groovy, happy, heavy,
holiday, hopeful, inspiring, love, meditative, melancholic, melodic, motivational, movie, nature, party, positive,
powerful, relaxing, retro, romantic, sad, sexy, slow, soft, soundscape, space, sport, summer, trailer, travel, upbeat,
uplifting
Models:
⬇️ mtg_jamendo_moodtheme-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mtg_jamendo_moodtheme-discogs-effnet-1.pb") predictions = model(embeddings)
Instrumentation¶
MTG-Jamendo instrument¶
Multi-label classification using the instrument subset of the MTG-Jamendo Dataset (40 classes):
accordion, acousticbassguitar, acousticguitar, bass, beat, bell, bongo, brass, cello, clarinet, classicalguitar, computer,
doublebass, drummachine, drums, electricguitar, electricpiano, flute, guitar, harmonica, harp, horn, keyboard, oboe,
orchestra, organ, pad, percussion, piano, pipeorgan, rhodes, sampler, saxophone, strings, synthesizer, trombone, trumpet,
viola, violin, voice
Models:
⬇️ mtg_jamendo_instrument-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mtg_jamendo_instrument-discogs-effnet-1.pb") predictions = model(embeddings)
Music loop instrument role¶
Classification of music loops by their instrument role using the Freesound Loop Dataset (5 classes):
bass, chords, fx, melody, percussion
Models:
⬇️ fs_loop_ds-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="fs_loop_ds-msd-musicnn-1.pb", input="serving_default_model_Placeholder", output="PartitionedCall") predictions = model(embeddings)
Mood Acoustic¶
Music classification by type of sound (2 classes):
acoustic, non_acoustic
Models:
⬇️ mood_acoustic-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_acoustic-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_acoustic-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_acoustic-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_acoustic-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_acoustic-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_acoustic-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_acoustic-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Mood Electronic¶
Music classification by type of sound (2 classes):
electronic, non_electronic
Models:
⬇️ mood_electronic-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_electronic-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_electronic-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_electronic-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_electronic-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_electronic-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ mood_electronic-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="mood_electronic-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Voice/instrumental¶
Classification of music by presence or absence of voice (2 classes):
instrumental, voice
Models:
⬇️ voice_instrumental-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="voice_instrumental-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ voice_instrumental-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="voice_instrumental-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ voice_instrumental-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="voice_instrumental-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ voice_instrumental-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="voice_instrumental-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ voice_instrumental-openl3-music-mel128-emb512
We do not have a dedicated algorithm to extract embeddings with this model. For now, OpenL3 embeddings can be extracted using this script.
Voice gender¶
Classification of music by singing voice gender (2 classes):
female, male
Models:
⬇️ gender-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="gender-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ gender-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="gender-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ gender-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="gender-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ gender-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="gender-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Timbre¶
Classification of music by timbre color (2 classes):
bright, dark
Models:
⬇️ timbre-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="timbre-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)
Nsynth acoustic/electronic¶
Classification of monophonic sources into acoustic or electronic origin using the Nsynth dataset (2 classes):
acoustic, electronic
Models:
⬇️ nsynth_acoustic_electronic-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="nsynth_acoustic_electronic-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)
Nsynth bright/dark¶
Classification of monophonic sources by timbre color using the Nsynth dataset (2 classes):
bright, dark
Models:
⬇️ nsynth_bright_dark-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="nsynth_bright_dark-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)
Nsynth instrument¶
Classification of monophonic sources by instrument family using the Nsynth dataset (11 classes):
mallet, string, reed, guitar, synth_lead, vocal, bass, flute, keyboard, brass, organ
Models:
⬇️ nsynth_instrument-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="nsynth_instrument-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)
Nsynth reverb¶
Detection of reverb in monophonic sources using the Nsynth dataset (2 classes):
dry, wet
Models:
⬇️ nsynth_reverb-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="nsynth_reverb-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)
Tonality¶
Tonal/atonal¶
Music classification by tonality (2 classes):
atonal, tonal
Models:
⬇️ tonal_atonal-audioset-vggish
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-vggish-3.pb", output="model/vggish/embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="tonal_atonal-audioset-vggish-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ tonal_atonal-audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="tonal_atonal-audioset-yamnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ tonal_atonal-discogs-effnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictEffnetDiscogs, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictEffnetDiscogs(graphFilename="discogs-effnet-bs64-1.pb", output="PartitionedCall:1") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="tonal_atonal-discogs-effnet-1.pb", output="model/Softmax") predictions = model(embeddings)⬇️ tonal_atonal-msd-musicnn
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictMusiCNN, TensorflowPredict2D audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() embedding_model = TensorflowPredictMusiCNN(graphFilename="msd-musicnn-1.pb", output="model/dense/BiasAdd") embeddings = embedding_model(audio) model = TensorflowPredict2D(graphFilename="tonal_atonal-msd-musicnn-1.pb", output="model/Softmax") predictions = model(embeddings)
Audio event recognition¶
AudioSet-YAMNet¶
Audio event recognition (520 audio event classes):
Speech, Child speech, kid speaking, Conversation, Narration, monologue, Babbling, Speech synthesizer, Shout, Bellow, Whoop,
Yell, Children shouting, Screaming, Whispering, Laughter, Baby laughter, Giggle, Snicker, Belly laugh, Chuckle, chortle,
Crying, sobbing, Baby cry, infant cry, Whimper, Wail, moan, Sigh, Singing, Choir, Yodeling, Chant, Mantra, Child singing,
Synthetic singing, Rapping, Humming, Groan, Grunt, Whistling, Breathing, Wheeze, Snoring, Gasp, Pant, Snort, Cough, Throat
clearing, Sneeze, Sniff, Run, Shuffle, Walk, footsteps, Chewing, mastication, Biting, Gargling, Stomach rumble, Burping,
eructation, Hiccup, Fart, Hands, Finger snapping, Clapping, Heart sounds, heartbeat, Heart murmur, Cheering, Applause, Chatter,
Crowd, Hubbub, speech noise, speech babble, Children playing, Animal, Domestic animals, pets, Dog, Bark, Yip, Howl, Bow-wow,
Growling, Whimper (dog), Cat, Purr, Meow, Hiss, Caterwaul, Livestock, farm animals, working animals, Horse, Clip-clop, Neigh,
whinny, Cattle, bovinae, Moo, Cowbell, Pig, Oink, Goat, Bleat, Sheep, Fowl, Chicken, rooster, Cluck, Crowing,
cock-a-doodle-doo, Turkey, Gobble, Duck, Quack, Goose, Honk, Wild animals, Roaring cats (lions, tigers), Roar, Bird, Bird
vocalization, bird call, bird song, Chirp, tweet, Squawk, Pigeon, dove, Coo, Crow, Caw, Owl, Hoot, Bird flight, flapping wings,
Canidae, dogs, wolves, Rodents, rats, mice, Mouse, Patter, Insect, Cricket, Mosquito, Fly, housefly, Buzz, Bee, wasp, etc.,
Frog, Croak, Snake, Rattle, Whale vocalization, Music, Musical instrument, Plucked string instrument, Guitar, Electric guitar,
Bass guitar, Acoustic guitar, Steel guitar, slide guitar, Tapping (guitar technique), Strum, Banjo, Sitar, Mandolin, Zither,
Ukulele, Keyboard (musical), Piano, Electric piano, Organ, Electronic organ, Hammond organ, Synthesizer, Sampler, Harpsichord,
Percussion, Drum kit, Drum machine, Drum, Snare drum, Rimshot, Drum roll, Bass drum, Timpani, Tabla, Cymbal, Hi-hat, Wood
block, Tambourine, Rattle (instrument), Maraca, Gong, Tubular bells, Mallet percussion, Marimba, xylophone, Glockenspiel,
Vibraphone, Steelpan, Orchestra, Brass instrument, French horn, Trumpet, Trombone, Bowed string instrument, String section,
Violin, fiddle, Pizzicato, Cello, Double bass, Wind instrument, woodwind instrument, Flute, Saxophone, Clarinet, Harp, Bell,
Church bell, Jingle bell, Bicycle bell, Tuning fork, Chime, Wind chime, Change ringing (campanology), Harmonica, Accordion,
Bagpipes, Didgeridoo, Shofar, Theremin, Singing bowl, Scratching (performance technique), Pop music, Hip hop music, Beatboxing,
Rock music, Heavy metal, Punk rock, Grunge, Progressive rock, Rock and roll, Psychedelic rock, Rhythm and blues, Soul music,
Reggae, Country, Swing music, Bluegrass, Funk, Folk music, Middle Eastern music, Jazz, Disco, Classical music, Opera,
Electronic music, House music, Techno, Dubstep, Drum and bass, Electronica, Electronic dance music, Ambient music, Trance
music, Music of Latin America, Salsa music, Flamenco, Blues, Music for children, New-age music, Vocal music, A capella, Music
of Africa, Afrobeat, Christian music, Gospel music, Music of Asia, Carnatic music, Music of Bollywood, Ska, Traditional music,
Independent music, Song, Background music, Theme music, Jingle (music), Soundtrack music, Lullaby, Video game music, Christmas
music, Dance music, Wedding music, Happy music, Sad music, Tender music, Exciting music, Angry music, Scary music, Wind,
Rustling leaves, Wind noise (microphone), Thunderstorm, Thunder, Water, Rain, Raindrop, Rain on surface, Stream, Waterfall,
Ocean, Waves, surf, Steam, Gurgling, Fire, Crackle, Vehicle, Boat, Water vehicle, Sailboat, sailing ship, Rowboat, canoe,
kayak, Motorboat, speedboat, Ship, Motor vehicle (road), Car, Vehicle horn, car horn, honking, Toot, Car alarm, Power windows,
electric windows, Skidding, Tire squeal, Car passing by, Race car, auto racing, Truck, Air brake, Air horn, truck horn,
Reversing beeps, Ice cream truck, ice cream van, Bus, Emergency vehicle, Police car (siren), Ambulance (siren), Fire engine,
fire truck (siren), Motorcycle, Traffic noise, roadway noise, Rail transport, Train, Train whistle, Train horn, Railroad car,
train wagon, Train wheels squealing, Subway, metro, underground, Aircraft, Aircraft engine, Jet engine, Propeller, airscrew,
Helicopter, Fixed-wing aircraft, airplane, Bicycle, Skateboard, Engine, Light engine (high frequency), Dental drill, dentist's
drill, Lawn mower, Chainsaw, Medium engine (mid frequency), Heavy engine (low frequency), Engine knocking, Engine starting,
Idling, Accelerating, revving, vroom, Door, Doorbell, Ding-dong, Sliding door, Slam, Knock, Tap, Squeak, Cupboard open or
close, Drawer open or close, Dishes, pots, and pans, Cutlery, silverware, Chopping (food), Frying (food), Microwave oven,
Blender, Water tap, faucet, Sink (filling or washing), Bathtub (filling or washing), Hair dryer, Toilet flush, Toothbrush,
Electric toothbrush, Vacuum cleaner, Zipper (clothing), Keys jangling, Coin (dropping), Scissors, Electric shaver, electric
razor, Shuffling cards, Typing, Typewriter, Computer keyboard, Writing, Alarm, Telephone, Telephone bell ringing, Ringtone,
Telephone dialing, DTMF, Dial tone, Busy signal, Alarm clock, Siren, Civil defense siren, Buzzer, Smoke detector, smoke alarm,
Fire alarm, Foghorn, Whistle, Steam whistle, Mechanisms, Ratchet, pawl, Clock, Tick, Tick-tock, Gears, Pulleys, Sewing machine,
Mechanical fan, Air conditioning, Cash register, Printer, Camera, Single-lens reflex camera, Tools, Hammer, Jackhammer, Sawing,
Filing (rasp), Sanding, Power tool, Drill, Explosion, Gunshot, gunfire, Machine gun, Fusillade, Artillery fire, Cap gun,
Fireworks, Firecracker, Burst, pop, Eruption, Boom, Wood, Chop, Splinter, Crack, Glass, Chink, clink, Shatter, Liquid, Splash,
splatter, Slosh, Squish, Drip, Pour, Trickle, dribble, Gush, Fill (with liquid), Spray, Pump (liquid), Stir, Boiling, Sonar,
Arrow, Whoosh, swoosh, swish, Thump, thud, Thunk, Electronic tuner, Effects unit, Chorus effect, Basketball bounce, Bang, Slap,
smack, Whack, thwack, Smash, crash, Breaking, Bouncing, Whip, Flap, Scratch, Scrape, Rub, Roll, Crushing, Crumpling, crinkling,
Tearing, Beep, bleep, Ping, Ding, Clang, Squeal, Creak, Rustle, Whir, Clatter, Sizzle, Clicking, Clickety-clack, Rumble, Plop,
Jingle, tinkle, Hum, Zing, Boing, Crunch, Silence, Sine wave, Harmonic, Chirp tone, Sound effect, Pulse, Inside, small room,
Inside, large room or hall, Inside, public space, Outside, urban or manmade, Outside, rural or natural, Reverberation, Echo,
Noise, Environmental noise, Static, Mains hum, Distortion, Sidetone, Cacophony, White noise, Pink noise, Throbbing, Vibration,
Television, Radio, Field recording
Models:
⬇️ audioset-yamnet
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictVGGish audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="activations") predictions = model(audio)Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictVGGish audio = MonoLoader(filename="audio.wav", sampleRate=16000, resampleQuality=4)() model = TensorflowPredictVGGish(graphFilename="audioset-yamnet-1.pb", input="melspectrogram", output="embeddings") embeddings = model(audio)
FSD-SINet¶
Audio event recognition using the FSD50K dataset targeting 200 classes drawn from the AudioSet Ontology.
The Shift Invariant Network (SINet) is offered in two different model sizes. vgg42 is a variation of vgg41 with twice the number of filters for each convolutional layer.
Also, the shift-invariance technique may be trainable low-pass filters (tlpf), adaptative polyphase sampling (aps), or both (tlpf_aps):
Accelerating and revving and vroom, Accordion, Acoustic guitar, Aircraft, Alarm, Animal, Applause, Bark, Bass drum, Bass
guitar, Bathtub (filling or washing), Bell, Bicycle, Bicycle bell, Bird, Bird vocalization and bird call and bird song, Boat
and Water vehicle, Boiling, Boom, Bowed string instrument, Brass instrument, Breathing, Burping and eructation, Bus, Buzz,
Camera, Car, Car passing by, Cat, Chatter, Cheering, Chewing and mastication, Chicken and rooster, Child speech and kid
speaking, Chime, Chink and clink, Chirp and tweet, Chuckle and chortle, Church bell, Clapping, Clock, Coin (dropping), Computer
keyboard, Conversation, Cough, Cowbell, Crack, Crackle, Crash cymbal, Cricket, Crow, Crowd, Crumpling and crinkling, Crushing,
Crying and sobbing, Cupboard open or close, Cutlery and silverware, Cymbal, Dishes and pots and pans, Dog, Domestic animals and
pets, Domestic sounds and home sounds, Door, Doorbell, Drawer open or close, Drill, Drip, Drum, Drum kit, Electric guitar,
Engine, Engine starting, Explosion, Fart, Female singing, Female speech and woman speaking, Fill (with liquid), Finger
snapping, Fire, Fireworks, Fixed-wing aircraft and airplane, Fowl, Frog, Frying (food), Gasp, Giggle, Glass, Glockenspiel,
Gong, Growling, Guitar, Gull and seagull, Gunshot and gunfire, Gurgling, Hammer, Hands, Harmonica, Harp, Hi-hat, Hiss, Human
group actions, Human voice, Idling, Insect, Keyboard (musical), Keys jangling, Knock, Laughter, Liquid, Livestock and farm
animals and working animals, Male singing, Male speech and man speaking, Mallet percussion, Marimba and xylophone, Mechanical
fan, Mechanisms, Meow, Microwave oven, Motor vehicle (road), Motorcycle, Music, Musical instrument, Ocean, Organ, Packing tape
and duct tape, Percussion, Piano, Plucked string instrument, Pour, Power tool, Printer, Purr, Race car and auto racing, Rail
transport, Rain, Raindrop, Ratchet and pawl, Rattle, Rattle (instrument), Respiratory sounds, Ringtone, Run, Sawing, Scissors,
Scratching (performance technique), Screaming, Screech, Shatter, Shout, Sigh, Singing, Sink (filling or washing), Siren,
Skateboard, Slam, Sliding door, Snare drum, Sneeze, Speech, Speech synthesizer, Splash and splatter, Squeak, Stream, Strum,
Subway and metro and underground, Tabla, Tambourine, Tap, Tearing, Telephone, Thump and thud, Thunder, Thunderstorm, Tick,
Tick-tock, Toilet flush, Tools, Traffic noise and roadway noise, Train, Trickle and dribble, Truck, Trumpet, Typewriter,
Typing, Vehicle, Vehicle horn and car horn and honking, Walk and footsteps, Water, Water tap and faucet, Waves and surf,
Whispering, Whoosh and swoosh and swish, Wild animals, Wind, Wind chime, Wind instrument and woodwind instrument, Wood,
Writing, Yell, Zipper (clothing)
Models:
⬇️ fsd-sinet-vgg41-tlpf
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg41-tlpf-1.pb") predictions = model(audio)Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg41-tlpf-1.pb", output="model/global_max_pooling1d/Max") embeddings = model(audio)⬇️ fsd-sinet-vgg42-aps
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-aps-1.pb") predictions = model(audio)Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-aps-1.pb", output="model/global_max_pooling1d/Max") embeddings = model(audio)⬇️ fsd-sinet-vgg42-tlpf_aps
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-tlpf_aps-1.pb") predictions = model(audio)Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-tlpf_aps-1.pb", output="model/global_max_pooling1d/Max") embeddings = model(audio)⬇️ fsd-sinet-vgg42-tlpf
Python code for predictions:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-tlpf-1.pb") predictions = model(audio)Python code for embedding extraction:
from essentia.standard import MonoLoader, TensorflowPredictFSDSINet audio = MonoLoader(filename="audio.wav", sampleRate=22050, resampleQuality=4)() model = TensorflowPredictFSDSINet(graphFilename="fsd-sinet-vgg42-tlpf-1.pb", output="model/global_max_pooling1d/Max") embeddings = model(audio)
Pitch detection¶
CREPE¶
Monophonic pitch detection (360 20-cent pitch bins, C1-B7) trained on the RWC-synth and the MDB-stem-synth datasets.
CREPE is offered with different model sizes ranging from tiny to full. A larger model is expected to perform better at the expense of additional computational costs.
Models:
Source separation¶
Spleeter¶
Source separation into 2, 4, or 5 stems.
Spleeter can separate music in different numbers of stems: 2 (vocals and accompaniment), 4 (vocals, drums, bass, and other separation), or 5 (vocals, drums, bass, piano, and other separation).
Models:
⬇️ speeter-2s
Python code for source separation:
from essentia.standard import AudioLoader, TensorflowPredict from essentia import Pool import numpy as np # Input should be audio @41kHz. audio, sr, _, _, _, _ = AudioLoader(filename="audio.wav")() pool = Pool() # The input needs to have 4 dimensions so that it is interpreted as an Essentia tensor. pool.set("waveform", audio[..., np.newaxis, np.newaxis]) model = TensorflowPredict( graphFilename="spleeter-2s-3.pb", inputs=["waveform"], outputs=["waveform_vocals", "waveform_accompaniment"] ) out_pool = model(pool) vocals = out_pool["waveform_vocals"].squeeze() accompaniment = out_pool["waveform_accompaniment"].squeeze()⬇️ speeter-4s
Python code for source separation:
from essentia.standard import AudioLoader, TensorflowPredict from essentia import Pool import numpy as np # Input should be audio @41kHz. audio, sr, _, _, _, _ = AudioLoader(filename="audio.wav")() pool = Pool() # The input needs to have 4 dimensions so that it is interpreted as an Essentia tensor. pool.set("waveform", audio[..., np.newaxis, np.newaxis]) model = TensorflowPredict( graphFilename="spleeter-4s-3.pb", inputs=["waveform"], outputs=["waveform_vocals", "waveform_drums", "waveform_bass", "waveform_other"] ) out_pool = model(pool) vocals = out_pool["waveform_vocals"].squeeze() drums = out_pool["waveform_drums"].squeeze() bass = out_pool["waveform_bass"].squeeze() other = out_pool["waveform_other"].squeeze()⬇️ speeter-5s
Python code for source separation:
from essentia.standard import AudioLoader, TensorflowPredict from essentia import Pool import numpy as np # Input should be audio @41kHz. audio, sr, _, _, _, _ = AudioLoader(filename="audio.wav")() pool = Pool() # The input needs to have 4 dimensions so that it is interpreted as an Essentia tensor. pool.set("waveform", audio[..., np.newaxis, np.newaxis]) model = TensorflowPredict( graphFilename="spleeter-5s-3.pb", inputs=["waveform"], outputs=["waveform_vocals", "waveform_drums", "waveform_bass", "waveform_piano", "waveform_other"] ) out_pool = model(pool) vocals = out_pool["waveform_vocals"].squeeze() drums = out_pool["waveform_drums"].squeeze() bass = out_pool["waveform_bass"].squeeze() bass = out_pool["waveform_piano"].squeeze() other = out_pool["waveform_other"].squeeze()
Tempo estimation¶
TempoCNN¶
Tempo classification (256 BPM classes, 30-286 BPM) trained on the Extended Ballroom, LMDTempo, and MTGTempo datasets.
TempoCNN may feature square filters (deepsquare) or longitudinal ones (deeptemp) and a model size factor of 4 (k4) or 16 (k16).
A larger model is expected to perform better at the expense of additional computational costs.
Models:
⬇️ deepsquare-k16
Python code for predictions:
from essentia.standard import MonoLoader, TempoCNN audio = MonoLoader(filename="audio.wav", sampleRate=11025, resampleQuality=4)() model = TempoCNN(graphFilename="deepsquare-k16-3.pb") global_tempo, local_tempo, local_tempo_probabilities = model(audio)