Deepfake Audio Detection for Social Engineering Defense

Abstract

The rapid advancement of speech synthesis has enabled audio deepfakes — synthetic voices virtually indistinguishable from genuine human speech. When exploited through social engineering attacks such as vishing and impersonation, they pose a severe threat to organisational security, as demonstrated by fraud incidents exceeding $25 million in losses. This thesis proposes a hybrid multi-modal detection architecture that simultaneously exploits three complementary acoustic features: Mel-Frequency Cepstral Coefficients (MFCC), Mel-Spectrogram, and Fundamental Frequency (F0), each processed by a dedicated convolutional branch. The resulting embeddings are combined through a learnable weighted fusion layer, refined by a two-layer Transformer encoder with multi-head self-attention, and classified by a four-layer MLP head. Evaluated on ASVspoof 2019 LA and WaveFake, the system achieves an EER of 2.03% and 0.0452% respectively, outperforming RawNet2 (5.13%) and LCNN (5.06%), and remaining competitive with AASIST (0.83%) at 7.8 lower computational cost (1.233 GFLOPs vs. 9.618 GFLOPs). A mixed training strategy (70% ASVspoof + 30% WaveFake) reduces the cross-dataset generalisation gap from 46.41% to 1.11% EER on WaveFake.