Hybrid visual–forensic deepfake detection

The question

Most deepfake detectors look only at pixels with a deep CNN. They score very well on the data they were trained on, but the assumption I set out to test is that they break down when an image is re-compressed (as happens on every social platform) or when the fake comes from a dataset they have never seen. My dissertation asks whether adding a second stream of hand-crafted forensic evidence, and letting the model learn how much to trust it, makes detection more reliable in exactly those conditions.

How it works

The detector takes a single face image and outputs the probability that it is fake.

• Visual stream: a pretrained EfficientNet-B4 (Noisy Student weights, via timm) producing a 1,792-dimensional embedding.
• Forensic stream: a fixed 406-dimensional feature vector computed from the pixels: error level analysis (262), DCT coefficient histograms (48), noise residual statistics (7), JPEG blockiness (4), an FFT radial spectrum (64), facial landmark symmetry (6) and colour statistics (15), passed through a small MLP.
• Attention-gated fusion: both embeddings are projected to 128 dimensions and a gate network outputs a softmax weight for each stream, per image, so the model can lean on forensic evidence when the pixels are unreliable. Concatenation and bilinear fusion are implemented as ablations.

Training runs in three phases (fusion only, then the forensic stream with the fusion, then full fine-tuning with a much lower learning rate for the backbone) using AdamW, a cosine schedule, mixed precision and early stopping on validation AUC. The project started as an earlier prototype that packed these signals into a padded 128-dimensional "metadata" vector; the final version replaced that with the fixed 406-feature layout, added identity-leakage checks and brought in new test sets.

Data and evaluation

I trained on FaceForensics++ (c23): 500 videos in each of five categories (original, Deepfakes, Face2Face, FaceSwap and NeuralTextures), giving 49,996 aligned 224-pixel face crops. Splits follow the official identity-disjoint lists, and the pipeline asserts that no identity crosses train, validation and test. Celeb-DF-v2 (5,177 test crops) was held out as an unseen dataset. Models are compared on AUC, equal error rate and partial AUC.

Findings

On the in-distribution test set the hybrid was the strongest model (AUC 0.981, equal error rate 7.1%), but its margin over the visual CNN alone was small. The compression sweep is where the results became interesting, and not in the direction I expected.

• The visual CNN proved fairly robust: its AUC fell gradually from 0.972 on clean images to 0.864 at JPEG quality 15.
• The forensic stream, trained on clean images, collapsed to near chance from quality 75 downwards, because compression destroys the very traces it measures.
• The attention gate responded sensibly: the weight it gave the forensic stream rose from 5% on clean images to 21% at quality 15.

Two of my starting premises were therefore not supported as stated. I treated that as a result rather than a failure, and used it to drive a second iteration.

Iteration two: training for robustness

For v2 I retrained with heavier compression and degradation augmentation (re-encoding, downscaling, blur, gamma and affine changes) and added StyleGAN-generated faces to the training data. Measured on the same FaceForensics++ test crops, the forensic stream held up far better under compression, at a small cost on clean images.

On Celeb-DF-v2 the hybrid's cross-dataset AUC improved from 0.726 to 0.783. The visual stream alone (0.791) edged it out, though, which is an honest limit of the fusion approach as it stands.

I also built a Gradio interface that detects and aligns the face in an uploaded image and shows the fake probability, each stream's vote, the gate weights and an ELA heatmap.

What I would do next

• Benchmark the concatenation and bilinear fusion ablations, which are implemented but were not yet evaluated.
• Test on a further held-out dataset such as DFDC: StyleGAN faces became in-distribution in v2, so they can no longer serve as an unseen-generator test.
• Look for forensic signals that survive heavy compression, rather than ones that compression erases.