[Q3-2026] Fingerprint Diagnostics: Benchmarking Fingerbank Against General-Purpose LLMs - Part 3

Executive Summary

We continue our series benchmarking Fingerbank against general-purpose large language models for network device identification. In this round we expanded the model lineup to six LLMs — three from the Claude 5 family (Sonnet 5, Opus 5, Fable 5) and three from the GPT family (GPT sol, GPT terra, GPT luna) — and stress-tested all seven engines, including Fingerbank itself, across 186 fingerprinting scenarios. Each scenario presented raw network telemetry and asked every engine to identify the manufacturer and device type.

The Efficiency Frontier: Cost vs. Latency

When deploying a fingerprinting solution at scale, efficiency is paramount. Network traffic waits for no one, and processing millions of requests requires both speed and cost-effectiveness.

🔍 Click to enlarge

Figure 1 — Average latency against average cost per call. Up-and-left represents the ideal "faster and cheaper" quadrant.

Fingerbank answered each query in 725 ms on average, at a cost of $0.00042 per call — a total of $0.08 to run all 186 scenarios. The LLMs ranged from 7.3 seconds (GPT terra) to 23.5 seconds (Fable 5). At the cheap end, GPT luna averaged 10.8 s per call at $0.0079 — 19 times more expensive than Fingerbank. At the other extreme, Fable 5 cost $0.237 per call — 568 times more expensive — and took 32 times longer to respond.

Accuracy: The Score Illusion

Every engine in this benchmark returns a confidence score (0–100) alongside its identification. On that metric, the GPT models appear dominant: GPT luna averaged 78.0, GPT sol 77.0, GPT terra 71.0. Fingerbank averaged just 40.3. But high scores and correct answers are not the same thing. When measured against ground-truth manufacturer labels, the ranking inverts entirely.

🔍 Click to enlarge

Figure 2 — Confidence vs. accuracy. GPT models are systematically overconfident; Claude models are conservatively calibrated.

Overall Accuracy and Manufacturer Match Rate

Fingerbank continues to significantly outperform all LLMs in both overall accuracy and manufacturer match rate:

Note on Confidence Score: for Fingerbank, the confidence score reflects the weight of signals that contributed to the identification; for LLMs, it is each model’s self-reported confidence. Fingerbank achieved an overall accuracy of 84.9%, maintaining a decisive lead over every evaluated LLM. Among the LLMs, Opus 5 posts the highest overall accuracy at 68.8%, followed by Sonnet 5 (66.1%) and Fable 5 (64.5%). The confidence-accuracy gap remains the critical finding for LLMs: GPT luna reports the highest self-rated confidence of any engine (78/100) while its accuracy trails Fingerbank by nearly 40 points.

Claude vs. GPT: Different Failure Modes

Claude models (Sonnet 5, Opus 5, Fable 5) achieve better correctness — Opus 5 leads all LLMs at 68.8% — but at substantially higher cost and latency. Fable 5 underperforms both sibling models on correctness (64.5%) while being the most expensive engine in the benchmark by a wide margin ($0.237/call). Among Claude models, Sonnet 5 represents the best cost-accuracy trade-off: correctness within 2.7 points of Opus 5 at 69% of the cost.

GPT models score higher but identify fewer manufacturers correctly. Their overconfidence is consistent across device categories — GPT terra, for instance, scored 71.0 on average while matching ground truth only 40.9% of the time. When telemetry signals are ambiguous, these models fill in gaps confidently rather than deferring. GPT sol is the notable outlier: correctness at 63.4% approaches the Claude tier, while cost ($0.033/call) makes it the most balanced LLM option when richer textual reasoning is required.

Why Fingerbank Wins: Structural Advantages

Raw accuracy numbers tell part of the story. The deeper reason Fingerbank outperforms general-purpose LLMs is architectural: it was designed from the ground up to solve exactly this problem, while LLMs were designed to solve everything.

Where Each Engine Excels — and Breaks Down

Device category results reveal that no single engine dominates uniformly. The variation across categories is substantial.

🔍 Click to interact

Figure 3 — Per-test consensus. Each column is a test case sorted by category; rows are engines. Green = correct, red = incorrect.

Medical Devices (n=6) produced the starkest gap. Fingerbank identified all six correctly (100%). LLMs ranged from 17% (Sonnet 5, Opus 5, GPT terra) to 50% (GPT sol). Specialized biomedical hardware sits far outside general-purpose training distributions.

Firewalls (n=11) showed a similar pattern. Fingerbank went 11 for 11 (100%). LLMs peaked at 64% (Opus 5, GPT sol, Fable 5). Security appliances from niche vendors rarely appear in web-scale corpora with the specificity needed to resolve DHCP option sets to a correct manufacturer string.

Storage Devices (n=13) was the notable equalizer. Fingerbank, GPT sol, Opus 5, and Sonnet 5 all hit 92% — NAS appliances carry distinctive OUI prefixes and well-documented DHCP patterns that every engine handles reliably.

VoIP and Conferencing (n=17) saw strong LLM performance: Sonnet 5, Opus 5, and Fable 5 each hit 82%, approaching Fingerbank's 88%. SIP devices from major vendors appear frequently in networking documentation that feeds LLM training data.

Signal Importance: What Telemetry Actually Helps

Not all network signals contribute equally. We tracked correctness separately for each case where a given signal was present, across all engines.

🔍 Click to interact

Figure 4 — Signal importance, ranked. Hostname and DHCP vendor are the richest signals; JA3 fingerprints and TCP SYN are hardest.

Hostname and DHCP vendor are the richest individual signals — when present, they push average accuracy to 76%. Hostnames encode vendor and model information that even general-purpose LLMs can decode reliably.

JA3 fingerprints and TCP signatures are the hardest. Resolving JA3 hashes to a manufacturer requires a lookup database that general-purpose models do not internalize from training data. Fingerbank's JA3 accuracy (83.7%) reflects purpose-built coverage; LLM accuracy on these cases ranged from 37% to 60%.

OUI is the most common signal (present in 179 of 186 cases) but sits in the middle tier at 62% average accuracy. The assignment-to-device-category inference required in ambiguous cases degrades performance for all engines.

Conclusion

Despite the maturity of both Claude 5 and the latest GPT generation for network device fingerprinting, a specialized engine like Fingerbank remains vastly superior to general-purpose LLMs. Fingerbank's 84.9% correctness rate exceeds the best LLM result (Opus 5 at 68.8%) by 16 percentage points — while being 21× faster and up to 568× cheaper per call. Three benchmarks into this series, the gap has not narrowed. If anything, it has clarified.

The GPT models' strong average scores (71–78) represent a calibration problem, not a capability advantage. GPT luna scores 78 while being correct only 45% of the time — a +32.8 point overconfidence gap. Operators who use the score field as a proxy for trust will systematically favour the engine with the worst actual accuracy. Correctness must always be measured against ground truth.

Real-time Performance: Significantly faster response times essential for network operations.

Cost Efficiency: Highly scalable operations at a fraction of the cost.

Superior Accuracy: Consistently higher accuracy and reliability, avoiding the confident hallucinations and high failure rates observed in LLMs.

View full diagnostic results