Essentia 2.1-dev Documentation

What is Essentia?

Essentia is an open-source C++ library with Python bindings for audio analysis and audio-based music information retrieval. It is released under the Affero GPLv3 license and is also available under proprietary license upon request. The library contains an extensive collection of reusable algorithms which implement audio input/output functionality, standard digital signal processing blocks, statistical characterization of data, and a large set of spectral, temporal, tonal and high-level music descriptors. In addition, Essentia can be complemented with Gaia, a C++ library with python bindings which implement similarity measures and classification on the results of audio analysis, and generate classification models that Essentia can use to compute high-level description of music (same license terms apply).

Essentia is not a framework, but rather a collection of algorithms (plus some infrastructure) wrapped in a library. It is designed with a focus on the robustness, performance and optimality of the provided algorithms, including computational speed and memory usage, as well as ease of use. The flow of the analysis is decided and implemented by the user, while Essentia is taking care of the implementation details of the algorithms being used. There is a special streaming mode in which it is possible to connect algorithms and run them automatically (similarly to PureData or Max/MSP) instead of specifying explicitly the order of execution with an advantage of less boilerplate code and less memory consumption. A number of examples are provided with the library, however they should not be considered as the only correct way of doing things. A large part of Essentia's algorithms is well-suited for real-time applications.

The provided functionality is easily expandable and allows for both research experiments and development of large-scale industrial applications. Essentia has served in a large number of research activities conducted at Music Technology Group since 2006. It has been used for music classification, semantic autotagging, music similarity and recommendation, visualization and interaction with music, sound indexing, musical instruments detection, cover detection, beat detection, and acoustic analysis of stimuli for neuroimaging studies. A list of highlighted academic publications can be found here. Essentia and Gaia have been used extensively in a number of research projects and industrial applications.

Currently the following algorithms are included (among others):

  • Audio file input/output: ability to read and write nearly all audio file formats (wav, mp3, ogg, flac, etc.)
  • Standard signal processing blocks: FFT, DCT, frame cutter, windowing, envelope, smoothing
  • Filters (FIR & IIR): low/high/band pass, band reject, DC removal, equal loudness
  • Statistical descriptors: median, mean, variance, power means, raw and central moments, spread, kurtosis, skewness, flatness
  • Time-domain descriptors: duration, loudness, LARM, Leq, Vickers' loudness, zero-crossing-rate, log attack time and other signal envelope descriptors
  • Spectral descriptors: Bark/Mel/ERB bands, MFCC, GFCC, LPC, spectral peaks, complexity, rolloff, contrast, HFC, inharmonicity and dissonance
  • Tonal descriptors: Pitch salience function, predominant melody and pitch, HPCP (chroma) related features, chords, key and scale, tuning frequency
  • Rhythm descriptors: beat detection, BPM, onset detection, rhythm transform, beat loudness
  • Other high-level descriptors: danceability, dynamic complexity, audio segmentation, SVM classifier

The library is cross-platform and currently supports Linux, Mac OS X, and partially Windows, iOS and Android systems. It can also be cross-compiled to JavaScript to be used on the web.

The library is wrapped in Python (Linux and OSX) and includes a number of predefined command-line extractors for music descriptors (Linux, OSX and Windows), which facilitates its use for fast prototyping and allows setting up research experiments very rapidly. Furthermore, it includes a Vamp plugin (Linux and OSX) that can be used with Sonic Visualiser for visualization purposes. There have been developed a number of third-party extensions to Essentia that allow its use within the frameworks of PureData and Max/MSP, openFrameworks, and Matlab.

Crediting Essentia

Please credit properly your use of Essentia! If you use the Essentia library in your software please acknowledge it and specify its origen as http://essentia.upf.edu. If you do some research and publish an article, cite both the Essentia paper [1] and the specific references mentioned in the documentation of the algorithms used. We would be also very grateful if you let us know how you use Essentia by sending an email to mtg@upf.edu

[1] Bogdanov, D., Wack N., Gómez E., Gulati S., Herrera P., Mayor O., et al. (2013). ESSENTIA: an Audio Analysis Library for Music Information Retrieval. International Society for Music Information Retrieval Conference (ISMIR'13). 493-498.

Contents

Getting started

Using Essentia

Extending Essentia

Download

Essentia releases | Gaia releases | Precompiled extractor binaries (Linux, OSX, Win)