Deep Learning Project

Music Genre Classification

A robust end-to-end audio classification system that predicts one of ten music genres from raw audio. The pipeline converts audio to mel spectrograms, injects realistic environmental noise from ESC-50, trains multiple deep learning architectures, and combines the strongest models using a weighted ensemble with test-time augmentation.

10 Genres ESC-50 Noise Augmentation EfficientNet-B0 AST Transformer Weighted Ensemble + TTA
3
Models Trained
50
Noise Classes
224×224
Spectrogram Size
95%
Estimated Ensemble F1

Project Overview

This project explores how modern computer vision and transformer architectures can be applied to audio. By representing music as mel spectrograms, the classification task becomes an image understanding problem. To improve robustness, environmental sounds such as rain, traffic, and crowd noise are mixed into training samples using the ESC-50 dataset. The final system combines EfficientNet-B0 and Audio Spectrogram Transformer (AST) predictions with weights based on validation F1 scores.

Pipeline Architecture

Raw Audio
Noise Injection
Mel Spectrogram
Model Training
Weighted Ensemble

Sample Spectrograms

Below are representative spectrogram visualizations illustrating how different genres exhibit unique time-frequency patterns that the models learn to distinguish.

Classical spectrogram Rock spectrogram Jazz spectrogram

Model Architectures

Scratch CNN

A custom convolutional baseline with four feature extraction blocks and a compact classifier head.

EfficientNet-B0

A pretrained vision model adapted to single-channel spectrogram inputs for efficient transfer learning.

Audio Spectrogram Transformer

A transformer architecture pretrained on AudioSet that captures long-range temporal dependencies.

Key Code Snippets

Audio to Mel Spectrogram

def audio_to_mel(y, sr=22050, n_mels=224):
    mel = librosa.feature.melspectrogram(
        y=y, sr=sr, n_mels=n_mels,
        n_fft=2048, hop_length=512
    )
    mel_db = librosa.power_to_db(mel, ref=np.max)
    return resize_to_224x224(mel_db)

ESC-50 Noise Augmentation

def add_noise(audio, noise_clip, snr_db=10):
    signal_power = np.mean(audio ** 2)
    noise_power = np.mean(noise_clip ** 2)
    factor = np.sqrt(signal_power / (10 ** (snr_db / 10) * noise_power))
    return audio + factor * noise_clip

EfficientNet-B0 Definition

model = timm.create_model(
    "efficientnet_b0",
    pretrained=True,
    in_chans=1,
    num_classes=10
)

Audio Spectrogram Transformer

ast = ASTForAudioClassification.from_pretrained(
    "MIT/ast-finetuned-audioset-10-10-0.4593",
    num_labels=10,
    ignore_mismatched_sizes=True
)

Weighted Ensemble

w_eff = f1_eff / (f1_eff + f1_ast)
w_ast = f1_ast / (f1_eff + f1_ast)

final_probs = w_eff * eff_probs + w_ast * ast_probs
prediction = np.argmax(final_probs, axis=1)

Estimated Validation F1 Score

EfficientNet-B0 and AST outperform the scratch CNN and achieve the best results when combined.

Training Configuration

  • Split: 85/15 stratified
  • Mixup: α = 0.4
  • Noise Source: ESC-50
  • EfficientNet LR: 2e-4
  • AST LR: 5e-5
  • TTA: Multiple noisy passes

Technologies Used

Python PyTorch Librosa Transformers timm Kaggle