Zero-Shot Defect Detection System Using One-Class Learning

A zero-shot defect detection/Re-Identification POC

Most AI-based visual inspection pipelines run into the same wall on day one: to train a defect classifier, pictures of defects are needed, which, by definition, are rare. A proof-of-concept script we reviewed takes a different, increasingly popular route: skip the defect images entirely, and teach the model what "good" looks like instead. Anything that doesn't match the "good" benchmark gets flagged for further inspection.

The problem: manual inspection doesn't scale, and defect data doesn't exist

Two issues drive most defect detection projects:

• Manual inspection is slow, inconsistent, and misses subtle defects. Fatigue and human variability result in even vigilant inspectors letting defective pieces through.

• There are rarely enough labeled defect images to train a classifier. In a healthy production line, defects might be 1-2% of output — nowhere near enough to train a supervised model, and the existing defect types won't cover the ones that have not been seen yet.

The pipeline we're reviewing (built around the MVTec AD benchmark dataset) is designed around the second constraint. It only ever trains on pristine reference images. There is no defect label anywhere in the training loop. This "one-class" framing mirrors how the strongest published defect-detection techniques (SPADE, PaDiM, PatchCore) are structured, and it is the only practical approach when defective samples are scarce.

How the pipeline actually works

Summarized, the script does the following:

1. Extract semantic features from a pretrained backbone. It loads a ResNet-18 pretrained on ImageNet and hooks into layer1 — an early block that still preserves a reasonably high-resolution spatial grid while encoding texture and shape information. Every image (256×256, ImageNet-normalized) is pushed through the network once, and the layer1 activation map is cached.

2. Build a "pristine prototype." All the good reference images for a category are run through the extractor, and their feature maps are averaged into a single prototype tensor. This is the pipeline's working definition of "normal."

3. Score anomalies via cosine distance. A test image's feature map is compared to the prototype using per-pixel cosine similarity. 1 - similarity becomes the raw distance map, which is up sampled back to 256×256 and passed through a Gaussian blur (21×21 kernel) to smooth out single-pixel noise and better reflect the area of a defect rather than isolated outlier pixels. The blurred map's maximum value becomes the image's anomaly score.

4. Calibrate a decision threshold without ever seeing a defect. This is the most interesting part of the implementation. Rather than picking an arbitrary cutoff, to calibrate the threshold, a leave-one-out pass is performed over the pristine set: each good image is scored against a prototype built from the other good images, producing a distribution of "how anomalous does a normal image look" scores. The 96th percentile of that distribution becomes the threshold. It's a self-supervised calibration trick that avoids hardcoding a magic number.

5. Decide and visualize. If a test image's score exceeds the threshold, it's flagged TAMPERED; otherwise PASS. A four-panel plot (pristine reference, test image, heatmap, overlay) makes the decision inspectable rather than a black box.

A fair assessment of the pipeline:

Essential Features:

True one-class learning. No defect labels, no synthetic anomalies, no supervised fine-tuning — it genuinely only needs "good" images, which matches the real-world data constraint.

Localization, not just classification. The heatmap tells an inspector where the anomaly is, which is far more actionable on a production line than a bare pass/fail flag.

Self-calibrating threshold. Deriving the cutoff from the statistics of the pristine set itself is a reasonable way to avoid hand-tuned constants, and it's an improvement over hardcoding a fixed distance value.

Lightweight and fast. ResNet-18 is a fraction of the size of the deeper backbones typically used for anomaly detection, and only one intermediate layer's activations are needed — this keeps inference cheap.

Sensible engineering touches for a POC. Caching extracted features and prototypes to avoid redundant forward passes across repeated calibration runs, which matters since threshold calibration reprocesses the pristine set once per reference image.

Bottlenecks for scaling to a production ready prototype:

Single-layer, single-scale features.

layer1 alone captures low-level texture and edges well but has limited semantic depth. Small, structural, or context-dependent defects that manifest at a different scale can slip past a single early layer. Published approaches (like PaDiM and PatchCore) deliberately fuse features from multiple layers to catch anomalies at multiple scales.

A mean prototype assumes exact orientation and lighting.

Averaging all pristine feature maps into one prototype assumes every "good" image looks nearly identical at every spatial location. In practice, normal variation (lighting, angle, natural texture variance) differs across regions of an object. Modeling each spatial location as a distribution (mean and covariance, as PaDiM does) and using something like Mahalanobis distance — or keeping a memory bank of real patch features and doing nearest-neighbor lookups, as PatchCore does — captures that natural variation far better than a single average.

No alignment or registration step.

Because the comparison is done location-by-location on a fixed 256×256 grid, any translation, rotation, or scale difference between a test image and the pristine set will show up as a false anomaly, independent of whether an actual defect exists.

Validation only against "good" images.

The 96th-percentile calibration never checks itself against a single real defect. It tells you how anomalous normal images can look, but not whether that cutoff actually separates true defects from noise — a gap that only labeled defect data (which MVTec AD conveniently provides, but the pipeline doesn't currently use) can close.

Fixed hyperparameters.

The Gaussian blur kernel (21×21, sigma 4) and the 96th-percentile cutoff are constants applied identically across every product category, defect type, and image resolution.

No explicit device handling.

The model and tensors implicitly run on CPU. On a real production line with camera feeds arriving continuously, that's a throughput bottleneck.

A generic, non-domain-adapted backbone.

ResNet-18's ImageNet weights are trained to recognize everyday objects, not industrial surface textures. It's a strong general-purpose feature extractor, but a small amount of domain adaptation (or a backbone specifically pretrained for industrial imagery) tends to sharpen the anomaly signal further.

What it would take to implement zero-shot defect detection system?

• Fusing multi-layer features (e.g., layer1 + layer2 + layer3) so the anomaly map is sensitive to both fine texture and coarser structural defects.

• Moving from a mean prototype to a proper statistical or memory-bank model — a per-location Gaussian (PaDiM-style) or a coreset-subsampled nearest-neighbor bank (PatchCore-style) — to represent the real variability within "good" images.

• Adding an alignment/registration preprocessing step so anomaly scores reflect actual defects rather than pose or framing differences.

• Making the threshold defect-aware where labeled anomalies exist — even a small labeled validation set can be used to sanity-check or adjust the leave-one-out threshold rather than relying purely on the "good" distribution.

• Adding GPU support and batching, plus a persistent (disk- or database-backed) cache instead of in-memory dictionaries, for real-time throughput on a live line.

• Tracking experiments. Logging scores, thresholds, and evaluation metrics per run — with a tool like MLflow — turns a one-off script into something a team can iterate on and audit over time.

Conclusion

We do not need labeled defect data to build a useful anomaly detector, we need a good definition of "normal" and a way to measure distance from it. The feature-extraction-plus-cosine-distance approach, combined with a self-calibrating threshold and a visual heatmap, is a genuinely reasonable starting architecture, and it's the same family of techniques behind some of the strongest published industrial anomaly detectors.

It is still a Proof Of Concept: it hasn't been stress-tested against a full labeled dataset, its statistical model of "normal" is simplistic (a single averaged prototype), and it's missing the engineering scaffolding — configuration, device handling, persistent caching, experiment tracking — that separates a demo script from a system a manufacturing line can depend on. None of that is a flaw in the core idea; it's the expected gap between a proof of concept and a production pipeline.

To find out whether an approach like this is worth building out further, reach out to us or explore our work in AI for manufacturing to see how we help production teams move from manual inspection to automated, data-driven quality control.

FAQs

1. Do I need labeled defect images to build an AI defect detector?

No. One-class / anomaly-detection approaches like the one reviewed here only need images of "good" products. They learn what normal looks like and flag anything that deviates from it, which is exactly why they work well in manufacturing settings where defects are rare and hard to collect at scale.

2. Why use an early layer like layer1 instead of deeper layers of a CNN?

Early layers preserve more spatial resolution, which matters for localizing exactly where a defect is. Deeper layers encode more semantic meaning but at a much coarser spatial grid. Most production anomaly detectors fuse several layers together to get the benefits of both.

3. How do you pick a decision threshold without any labeled defect data?

One approach — used in this pipeline — is to calibrate against the "good" reference set itself: score each good image against a prototype built from the others, and use a high percentile of that distribution as the cutoff. It's a reasonable starting point, but it should ideally be validated against real defect examples where available, since it only tells you how anomalous normal images can look, not how well the cutoff separates true defects.