OpenL3 tutorial¶
Introduction¶
Welcome to the OpenL3 tutorial! In this tutorial, we’ll show how to use OpenL3
to compute embeddings for your audio files, images, and videos. The supported audio formats
are those supported by the pysoundfile
library, which is used for loading the
audio (e.g. WAV, OGG, FLAC). The supported image formats are those supported by
the scikit-image` library (e.g. PNG, JPEG). The supported video formats are those
supported by moviepy
(and therefore ffmpeg
) (e.g. MP4).
Using the Library¶
Extracting audio embeddings¶
You can easily compute audio embeddings out of the box, like so:
import openl3
import soundfile as sf
audio, sr = sf.read('/path/to/file.wav')
emb, ts = openl3.get_audio_embedding(audio, sr)
get_audio_embedding
returns two objects. The first object emb
is a T-by-D numpy array,
where T is the number of embedding frames and D is the dimensionality
of the embedding (which can be either 6144 or 512, details below). The second object ts
is a length-T
numpy array containing timestamps corresponding to each embedding frame (each timestamp corresponds
to the center of each analysis window by default).
Multiple audio arrays can be provided (with a specified model input batch size for processing, i.e. the number of audio windows the model processes at a time) as follows:
import openl3
import soundfile as sf
audio1, sr1 = sf.read('/path/to/file1.wav')
audio2, sr2 = sf.read('/path/to/file2.wav')
audio3, sr3 = sf.read('/path/to/file3.wav')
audio_list = [audio1, audio2, audio3]
sr_list = [sr1, sr2, sr3]
# Pass in a list of audio arrays and sample rates
emb_list, ts_list = openl3.get_audio_embedding(audio_list, sr_list, batch_size=32)
# If all arrays use sample rate, can just pass in one sample rate
emb_list, ts_list = openl3.get_audio_embedding(audio_list, sr1, batch_size=32)
Here, we get a list of embeddings and timestamp arrays for each of the input arrays.
By default, OpenL3 extracts audio embeddings with a model that:
Is trained on AudioSet videos containing mostly musical performances
Uses a mel-spectrogram time-frequency representation with 128 bands
Returns an embedding of dimensionality 6144 for each embedding frame
These defaults can be changed via the following optional parameters:
content_type: “env”, “music” (default)
input_repr: “linear”, “mel128” (default), “mel256”
embedding_size: 512, 6144 (default)
For example, the following code computes an embedding using a model trained on environmental videos using a spectrogram with a linear frequency axis and an embedding dimensionality of 512:
emb, ts = openl3.get_audio_embedding(audio, sr, content_type="env",
input_repr="linear", embedding_size=512)
By default OpenL3 will pad the beginning of the input audio signal by 0.5 seconds (half of the window size) so that the the center of the first window corresponds to the beginning of the signal (“zero centered”), and the returned timestamps correspond to the center of each window. You can disable this centering like this:
emb, ts = openl3.get_audio_embedding(audio, sr, center=False)
The hop size used to extract the embedding is 0.1 seconds by default (i.e. an embedding frame rate of 10 Hz). In the following example we change the hop size from 0.1 (10 frames per second) to 0.5 (2 frames per second):
emb, ts = openl3.get_audio_embedding(audio, sr, hop_size=0.5)
Finally, you can silence the Keras printout during inference (verbosity) by changing it from 1 (default) to 0:
emb, ts = openl3.get_audio_embedding(audio, sr, verbose=0)
By default, the model file is loaded from disk every time get_audio_embedding
is called. To avoid unnecessary I/O when
processing multiple files with the same model, you can load it manually and pass it to the function via the
model
parameter:
model = openl3.models.load_audio_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
emb1, ts1 = openl3.get_audio_embedding(audio1, sr1, model=model)
emb2, ts2 = openl3.get_audio_embedding(audio2, sr2, model=model)
Note that when a model is provided via the model
parameter any values passed to the input_repr
, content_type
and
embedding_size
parameters of get_audio_embedding
will be ignored.
To compute embeddings for an audio file and directly save them to disk you can use process_audio_file
:
import openl3
import numpy as np
audio_filepath = '/path/to/file.wav'
# Save the embedding to '/path/to/file.npz'
openl3.process_audio_file(audio_filepath)
# Save the embedding to `/path/to/file_suffix.npz`
openl3.process_audio_file(audio_filepath, suffix='suffix')
# Save the embedding to '/different/dir/file_suffix.npz'
openl3.process_audio_file(audio_filepath, suffix='suffix', output_dir='/different/dir')
The embddings can be loaded from disk using numpy:
import numpy as np
data = np.load('/path/to/file.npz')
emb, ts = data['embedding'], data['timestamps']
Multiple files can be processed as well (with a specified model input batch size used for processing, i.e. the number of audio windows the model processes at a time) as follows:
import openl3
import numpy as np
audio_filepath1 = '/path/to/file1.wav'
audio_filepath2 = '/path/to/file2.wav'
audio_filepath3 = '/path/to/file3.wav'
audio_filepath_list = [audio_filepath1, audio_filepath2, audio_filepath3]
# Saves embeddings to '/path/to/file1.npz', '/path/to/file2.npz', and '/path/to/file3.npz'
openl3.process_audio_file(audio_filepath_list, batch_size=32)
As with get_audio_embedding
, you can load the model manually and pass it to process_audio_file
to avoid loading the model multiple times:
import openl3
import numpy as np
model = openl3.models.load_audio_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
audio_filepath = '/path/to/file.wav'
# Save the file to '/path/to/file.npz'
openl3.process_audio_file(audio_filepath, model=model)
# Save the file to `/path/to/file_suffix.npz`
openl3.process_audio_file(audio_filepath, model=model, suffix='suffix')
# Save the file to '/different/dir/file_suffix.npz'
openl3.process_audio_file(audio_filepath, model=model, suffix='suffix', output_dir='/different/dir')
Again, note that if a model is provided via the model
parameter, then any values passed to the input_repr
, content_type
and embedding_size
parameters of process_audio_file
will be ignored.
Extracting image embeddings¶
You can easily compute image embeddings out of the box, like so:
import openl3
from skimage.io import imread
image = imread('/path/to/file.png)
emb = openl3.get_image_embedding(image, content_type="env",
input_repr="linear", embedding_size=512)
# Preload model
model = openl3.models.load_image_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
emb = openl3.get_image_embedding(image, model=model)
When given a single image, get_image_embedding
returns a size D numpy array,
where D is the dimensionality of the embedding (which can be either 8192 or 512).
A sequence of images (e.g. from a video) can also be provided:
from moviepy.video.io.VideoFileClip import VideoFileClip
video_filepath = '/path/to/file.mp4'
# Load video and get image frames
clip = VideoFileClip(video_filepath)
images = np.array([frame for frame in clip.iter_frames()])
emb = get_image_embedding(images, batch_size=32)
When given a sequence of images (as a numpy array), get_image_embedding
returns an N-by-D numpy array,
where N is the number of embedding frames (corresponding to each video frame) and D is the dimensionality
of the embedding (which can be either 8192 or 512).
from moviepy.video.io.VideoFileClip import VideoFileClip
video_filepath = '/path/to/file.mp4'
# Load video and get image frames
clip = VideoFileClip(video_filepath)
images = np.array([frame for frame in clip.iter_frames()])
# If the frame rate is provided, returns an array of timestamps
emb, ts = get_image_embedding(images, frame_rate=clip.fps, batch_size=32)
When given a sequence of images (as a numpy array) and a frame rate, get_image_embedding
returns two objects.
The first object emb
is an N-by-D numpy array, where N is the number of embedding frames and D is the dimensionality
of the embedding (which can be either 8192 or 512, details below). The second object ts
is a length-N
numpy array containing timestamps corresponding to each embedding frame (corresponding to each video frame).
Multiple sequences of images can be provided as well:
from moviepy.video.io.VideoFileClip import VideoFileClip
video_filepath1 = '/path/to/file1.mp4'
video_filepath2 = '/path/to/file2.mp4'
video_filepath3 = '/path/to/file2.mp4'
# Load video and get image frames
clip1 = VideoFileClip(video_filepath1)
images1 = np.array([frame for frame in clip1.iter_frames()])
clip2 = VideoFileClip(video_filepath2)
images2 = np.array([frame for frame in clip2.iter_frames()])
clip3 = VideoFileClip(video_filepath3)
images3 = np.array([frame for frame in clip3.iter_frames()])
image_list = [images1, images2, images3]
frame_rate_list = [clip1.fps, clip2.fps, clip3.fps]
# If the frame rates is provided...
emb_list, ts_list = get_image_embedding(image_list, frame_rate=frame_rate_list, batch_size=32)
# or if a single frame rate applying to all sequences is provided, returns a list of timestamps
emb_list, ts_list = get_image_embedding(image_list, frame_rate=clip1.fps, batch_size=32)
# ...otherwise, just the embeddings are returnedj
emb_list = get_image_embedding(image_list, batch_size=32)
Here, we get a list of embeddings for each sequence. If a frame rate (or list of frame rates) is given, timestamp arrays for each of the input arrays are returned as well.
By default, OpenL3 extracts image embeddings with a model that:
Is trained on AudioSet videos containing mostly musical performances
Is trained using a mel-spectrogram time-frequency representation with 128 bands (for the audio embedding model)
Returns an embedding of dimensionality 8192 for each embedding frame
These defaults can be changed via the following optional parameters:
content_type: “env”, “music” (default)
input_repr: “linear”, “mel128” (default), “mel256”
embedding_size: 512, 8192 (default)
For example, the following code computes an embedding using a model trained on environmental videos using an audio spectrogram with a linear frequency axis and an embedding dimensionality of 512:
emb = openl3.get_image_embedding(image, content_type="env",
input_repr="linear", embedding_size=512)
Finally, you can silence the Keras printout during inference (verbosity) by changing it from 1 (default) to 0:
emb = openl3.get_image_embedding(image, verbose=0)
By default, the model file is loaded from disk every time get_image_embedding
is called. To avoid unnecessary I/O when
processing multiple files with the same model, you can load it manually and pass it to the function via the
model
parameter:
model = openl3.models.load_image_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
emb1 = openl3.get_image_embedding(image1, model=model)
emb2 = openl3.get_image_embedding(image2, model=model)
Note that when a model is provided via the model
parameter any values passed to the input_repr
, content_type
and
embedding_size
parameters of get_image_embedding
will be ignored.
Image files can also be processed with just the filepath:
import openl3
import numpy as np
image_filepath = '/path/to/file.png'
# Save the file to '/path/to/file.npz'
openl3.process_image_file(image_filepath)
# Preload model
model = openl3.models.load_image_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
openl3.process_image_file(image_filepath, model=model)
# Save the file to `/path/to/file_suffix.npz`
openl3.process_image_file(image_filepath, model=model, suffix='suffix')
# Save the file to '/different/dir/file_suffix.npz'
openl3.process_image_file(image_filepath, model=model, suffix='suffix', output_dir='/different/dir')
The embdding can be loaded from disk using numpy:
import numpy as np
data = np.load('/path/to/file.npz')
emb = data['embedding']
Multiple files can be processed as well (with a specified model input batch size used for processing, i.e. the number of audio windows the model processes at a time) as follows:
import openl3
import numpy as np
image_filepath1 = '/path/to/file1.png'
image_filepath2 = '/path/to/file2.png'
image_filepath3 = '/path/to/file3.png'
image_filepath_list = [image_filepath1, image_filepath2, image_filepath3]
# Saves embeddings to '/path/to/file1.npz', '/path/to/file2.npz', and '/path/to/file3.npz'
openl3.process_image_file(image_filepath_list, batch_size=32)
As with get_image_embedding
, you can load the model manually and pass it to process_image_file
to avoid loading the model multiple times:
import openl3
import numpy as np
model = openl3.models.load_image_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
image_filepath = '/path/to/file.png'
# Save the file to '/path/to/file.npz'
openl3.process_image_file(image_filepath, model=model)
# Save the file to `/path/to/file_suffix.npz`
openl3.process_image_file(image_filepath, model=model, suffix='suffix')
# Save the file to '/different/dir/file_suffix.npz'
openl3.process_image_file(image_filepath, model=model, suffix='suffix', output_dir='/different/dir')
Again, note that if a model is provided via the model
parameter, then any values passed to the input_repr
, content_type
and embedding_size
parameters of process_image_file
will be ignored.
Processing video files¶
Video files can be processed to extract both audio and image embeddings. Please note that the audio and image embeddings are not synchronized, so the respective timestamps for each modality will not generally be aligned. Image embeddings are computed for every frame of the video, while the specified audio hop size is used for chunking the audio signal in the video. Additionally, please note that available embedding sizes differ for audio and image embeddings. Video files can be processed as follows:
import openl3
import numpy as np
video_filepath = '/path/to/file.mp4'
# Save audio embedding to '/path/to/file_audio.npz'
# and image embedding to '/path/to/file_image.npz'
openl3.process_video_file(video_filepath)
# Preload models
audio_model = openl3.models.load_audio_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
image_model = openl3.models.load_image_embedding_model(input_repr="mel256", content_type="music",
embedding_size=512)
openl3.process_video_file(video_filepath, audio_model=audio_model, image_model=image_model)
# Save audio embedding to '/path/to/file_audio_suffix.npz'
# and image embedding to '/path/to/file_image_suffix.npz'
openl3.process_video_file(video_filepath, audio_model=audio_model, image_model=image_model,
suffix='suffix')
# Save audio embedding to '/different/dir/file_audio_suffix.npz'
# and image embedding to '/different/dir/file_image_suffix.npz'
openl3.process_video_file(video_filepath, audio_model=audio_model, image_model=image_model,
suffix='suffix', output_dir='/different/dir')
Multiple files can be processed as well (with a specified model input batch size used for processing, i.e. the number of video frames the model processes at a time) as follows:
import openl3
import numpy as np
video_filepath1 = '/path/to/file1.mp4'
video_filepath2 = '/path/to/file2.mp4'
video_filepath3 = '/path/to/file3.mp4'
video_filepath_list = [video_filepath1, video_filepath2, video_filepath3]
# Saves audio embeddings to '/path/to/file1_audio.npz', '/path/to/file2_audio.npz',
# and '/path/to/file3_audio.npz' and saves image embeddings to
# '/path/to/file1_image.npz', '/path/to/file2_image.npz', and '/path/to/file3_image.npz'
openl3.process_video_file(video_filepath_list, batch_size=32)
Choosing an Audio Frontend (CPU / GPU)¶
OpenL3 provides two different audio frontends to choose from.
Kapre (GPU)¶
Kapre implements audio processing as neural network layers, and computes the audio front-end
(spectrogram or mel-spectrogram) as part of running model inference. This means that when
a GPU is available, audio processing happens on the GPU. The OpenL3 Keras model takes batches
of audio PCM as input (shape=(None, 1, samples))
.
NOTE: due to updates in Kapre, the embeddings produced by OpenL3>=0.4.0 are not identical to those produced by previous versions (e.g. 0.3.1), even though both use Kapre for audio processing. If you are running inference with a model that was trained on OpenL3 embeddings, we strongly recommend using the same version of OpenL3 that was used to train that model.
import openl3
import soundfile as sf
audio, sr = sf.read('/path/to/file1.wav')
# get embedding using kapre frontend
emb_list, ts_list = openl3.get_audio_embedding(audio, sr)
emb_list, ts_list = openl3.get_audio_embedding(audio, sr, frontend='kapre') # same result
# pre-loading a model
# load model with the kapre frontend
input_repr, content_type, embedding_size = 'mel128', 'music', 6144
model_kapre = openl3.models.load_audio_embedding_model(input_repr, content_type, embedding_size)
# get embedding using pre-loaded model with kapre frontend
emb_list, ts_list = openl3.get_audio_embedding(audio, sr, model=model_kapre)
# computing embeddings manually for a single file.
# Note: If no input representation (`input_repr`) is passed to `preprocess_audio`
# then it will prepare the input data for the kapre frontend.
input_data = openl3.preprocess_audio(audio, sr)
emb, ts = model_kapre.predict(input_data)
Librosa (CPU)¶
Beginning with version 0.4.0, OpenL3 also provides support for computing audio features using Librosa, which offers you flexibility and allows you to precompute or parallelize your computations across multiple CPUs.
Contrary to the Kapre frontend, the Librosa frontend is not included as part of the neural network model. Instead the Keras model takes either linear or mel spectrograms (according to the input representation chosen). OpenL3’s high-level interfaces still work the same though, it just changes the mechanics of how features are calculated under the hood.
This also decouples the feature extraction, which consists of specialized audio code that may not export well (e.g., to ONNX), from the standard convolutional and fully-connected layers that comprise the rest of OpenL3, making it easier to port the model to other neural network frameworks.
NOTE: input_repr
is a required parameter to openl3.get_audio_embedding
when using the librosa frontend, because the feature
extraction steps are no longer compiled into the model.
import openl3
import soundfile as sf
audio, sr = sf.read('/path/to/file1.wav')
# get embedding using librosa frontend
emb_list, ts_list = openl3.get_audio_embedding(audio, sr, frontend='librosa')
# pre-loading a model
# load model with no frontend for use with an external librosa frontend
input_repr, content_type, embedding_size = 'mel128', 'music', 6144
model = openl3.models.load_audio_embedding_model(
input_repr, content_type, embedding_size, frontend='librosa')
# get embedding using pre-loaded model and librosa frontend
emb_list, ts_list = openl3.get_audio_embedding(audio, sr, model=model, frontend='librosa')
# computing embeddings manually for a single file
# Note how we pass the input representation (`input_repr`) to `preprocess_audio` so that
# librosa knows how to compute the inputs.
input_data = openl3.preprocess_audio(audio, sr, input_repr=input_repr)
emb, ts = model.predict(input_data)
The same applies for video file processing, using audio_frontend='librosa'
# load model with no frontend for use with an external librosa frontend
input_repr, content_type, embedding_size = 'mel128', 'music', 6144
model = openl3.models.load_audio_embedding_model(
input_repr, content_type, embedding_size, frontend='librosa')
openl3.process_video_file(video_fname, audio_model=model, audio_frontend='librosa')
Using the Command Line Interface (CLI)¶
Extracting audio embeddings¶
To compute embeddings for a single audio file via the command line run:
$ openl3 audio /path/to/file.wav
This will create an output file at /path/to/file.npz
.
You can change the output directory as follows:
$ openl3 audio /path/to/file.wav --output /different/dir
This will create an output file at /different/dir/file.npz
.
You can also provide multiple input files:
$ openl3 audio /path/to/file1.wav /path/to/file2.wav /path/to/file3.wav
which will create the output files /different/dir/file1.npz
, /different/dir/file2.npz
,
and different/dir/file3.npz
.
You can also provide one (or more) directories to process:
$ openl3 audio /path/to/audio/dir
This will process all supported audio files in the directory, though it will not recursively traverse the directory (i.e. audio files in subfolders will not be processed).
You can append a suffix to the output file as follows:
$ openl3 audio /path/to/file.wav --suffix somesuffix
which will create the output file /path/to/file_somesuffix.npz
.
Arguments can also be provided to change the model used to extract the embedding including the content type used for training (music or env), input representation (linear, mel128, mel256), and output dimensionality (512 or 6144), for example:
$ openl3 audio /path/to/file.wav --content-type env --input-repr mel128 --embedding-size 512
The default value for –content-type is music, for –input-repr is mel128 and for –embedding-size is 6144.
By default, OpenL3 will pad the beginning of the input audio signal by 0.5 seconds (half of the window size) so that the the center of the first window corresponds to the beginning of the signal, and the timestamps correspond to the center of each window. You can disable this centering as follows:
$ openl3 audio /path/to/file.wav --no-audio-centering
The hop size used to extract the embedding is 0.1 seconds by default (i.e. an embedding frame rate of 10 Hz). In the following example we change the hop size from 0.1 (10 frames per second) to 0.5 (2 frames per second):
$ openl3 audio /path/to/file.wav --audio-hop-size 0.5
You can change the batch size used as the input to the audio embedding model (i.e. the number of audio windows the model processes at a time) to one appropriate for your computational resources:
$ openl3 audio /path/to/file.wav --audio-batch-size 16
By default, the CLI will use the Kapre frontend to compute audio embeddings. You may also choose to use Librosa to compute the embeddings:
$ openl3 audio /path/to/file.wav --audio-frontend librosa
Finally, you can suppress non-error printouts by running:
$ openl3 audio /path/to/file.wav --quiet
Extracting image embeddings¶
To compute embeddings for a single image file via the command line run:
$ openl3 image /path/to/file.png
This will create an output file at /path/to/file.npz
.
You can change the output directory as follows:
$ openl3 image /path/to/file.png --output /different/dir
This will create an output file at /different/dir/file.npz
.
You can also provide multiple input files:
$ openl3 image /path/to/file1.png /path/to/file2.png /path/to/file3.png
which will create the output files /different/dir/file1.npz
, /different/dir/file2.npz
,
and different/dir/file3.npz
.
You can also provide one (or more) directories to process:
$ openl3 image /path/to/image/dir
This will process all supported image files in the directory, though it will not recursively traverse the directory (i.e. image files in subfolders will not be processed).
You can append a suffix to the output file as follows:
$ openl3 image /path/to/file.png --suffix somesuffix
which will create the output file /path/to/file_somesuffix.npz
.
Arguments can also be provided to change the model used to extract the embedding including the content type used for training (music or env), input representation (linear, mel128, mel256), and output dimensionality (512 or 8192), for example:
$ openl3 image /path/to/file.png --content-type env --input-repr mel128 --embedding-size 512
The default value for –content-type is music, for –input-repr is mel128 and for –embedding-size is 8192.
You can change the batch size used as the input to the image embedding model (i.e. the number of video frames the model processes at a time) to one appropriate for your computational resources:
$ openl3 image /path/to/file.png --image-batch-size 16
Finally, you can suppress non-error printouts by running:
$ openl3 image /path/to/file.png --quiet
Processing video files¶
To compute embeddings for a single video file via the command line run:
$ openl3 video /path/to/file.mp4
This will create output files at /path/to/file_audio.npz
and /path/to/file_image.npz
for the audio and image embeddings, respectively. Please note that the audio and image embeddings are
not synchronized, so the respective timestamps for each modality will not generally be aligned.
Image embeddings are computed for every frame of the video, while the specified audio hop size is used
for chunking the audio signal in the video. Additionally, please note that available embedding sizes differ
for audio and image embeddings.
Functionality regarding specifying models, multiple input files, verbosity, output directories, and suffixes behave the same as with extracting audio and image embeddings. Functionality specific to audio or image embeddings can also be specified as previously specified.
You can change the batch size used as the input to the audio embedding and image embedding models (i.e. the number of audio windows or video frames, respectively, that the model processes at a time) to one appropriate for your computational resources:
$ openl3 video /path/to/file.mp4 --audio-batch-size 16 --image-batch-size 16