[Q3-2026] Fingerprint Diagnostics: Benchmarking Fingerbank Against General-Purpose LLMs - Part 2

We continue our experiments to answer the question: Can general-purpose Large Language Models (LLMs) replace specialized, purpose-built engines? We conducted a comprehensive benchmark comparing Fingerbank’s production fingerprinting engine against four LLMs: DeepSeek v4 Flash, DeepSeek v4 Pro, Kimi k2.5, and Kimi k2.6.

The benchmark evaluated each engine across 206 distinct scenarios, totaling 1,030 runs. We assessed them based on speed (latency), cost, self-reported confidence, and manufacturer accuracy using the exact same input telemetry—including MAC addresses, DHCP fingerprints, User-Agents, TCP signatures, and more.

The results are in, and they reinforce the enduring value of specialized engineering, even as a new wave of capable open-weight LLMs from DeepSeek and Moonshot AI enters the field.

When deploying a fingerprinting solution at scale, efficiency is paramount. Network traffic waits for no one, and processing millions of requests requires both speed and cost-effectiveness.

🔍 Click to enlarge

🔍 Click to enlarge

Figure 1: Average latency against average cost per call. Up-and-right represents the ideal “faster and cheaper” quadrant.

The data reveals a stark contrast between Fingerbank and the LLMs:

Fingerbank remains the leader in efficiency. At 488 ms, it is roughly 30 times faster than the quickest LLM tested. DeepSeek v4 Flash emerges as the fastest LLM at 14.73 seconds, and is also the cheapest of the group at $0.0005 per call—only marginally above Fingerbank’s $0.0004. At the other extreme, Kimi k2.6 is both the slowest (127.43 seconds, more than two minutes per call) and the most expensive ($0.0135 per call). The cost-per-call ranges further underscore the unpredictability of LLM pricing: while Fingerbank’s cost is flat at $0.0004, Kimi k2.6 swings from $0.0043 to $0.0752 depending on the scenario.

Speed and cost mean little if the identification is incorrect. We measured detection correctness across various device categories and evaluated how often each engine successfully matched the officially registered manufacturer.

🔍 Click to enlarge

Figure 2: Detection correctness by test category (left) and manufacturer match rate against Fingerbank’s OUI lookup (right).

Overall Accuracy and Manufacturer Match Rate

Fingerbank continues to significantly outperform all LLMs in both overall accuracy and manufacturer match rate:

Note on Confidence Score: For Fingerbank, the confidence score is calculated based on the weight of signals that participated in the detection process. For LLMs, it represents their self-estimated confidence.

Fingerbank achieved an overall accuracy of 85%, maintaining a commanding lead over every evaluated LLM. Among the LLMs, DeepSeek v4 Flash leads at 47%, followed by DeepSeek v4 Pro at 41%, while the Kimi models trail at 33–35%. The confidence-accuracy gap remains a critical observation: DeepSeek v4 Pro reports the highest confidence (71/100) yet identifies fewer than half of devices correctly, whereas Fingerbank pairs the lowest self-reported confidence (39/100) with by far the highest accuracy—a sign of well-calibrated, signal-grounded scoring rather than optimistic guessing.

Category Breakdown

Fingerbank’s specialized knowledge continues to shine across device categories:

Gaming Console and Firewall: Fingerbank achieved 100% accuracy in these categories, demonstrating its reliability where precision is paramount. The best LLMs reached only 50–82% here.

Medical Devices, Physical Security, and Robotics: Fingerbank scored 89%, 91%, and 85% respectively, while LLMs collapsed to as low as 9–11%—precisely the high-stakes categories where confident hallucination is most dangerous.

Storage Devices and Automotive, Energy and Tools: These remain challenging for LLMs, with most scoring 17–27%, compared to Fingerbank’s 78% and 87%.

Device fingerprinting relies on combining multiple signals (OUI, DHCP, mDNS, User-Agent, etc.) to form a conclusive identity. We analyzed how detection correctness varies based on which signals appear together in the payload.

🔍 Click to interact

Figure 3: Co-occurrence heatmap showing Fingerbank’s accuracy based on signal combinations.

The heatmap emphasizes Fingerbank’s robustness and its ability to synthesize complex signal combinations effectively. It consistently maintains high accuracy across diverse signal pairings—reaching 100% when mDNS services co-occur with DHCP, JA3, TCP, or UPnP signals. LLMs, while capable of parsing individual signals, often struggle with the nuanced interpretation and weighting required for accurate device identification when multiple or conflicting signals are present.

Beyond accuracy, the failure rate—how often an engine returned a device name containing “error” or “unknown”—is crucial for real-world deployments. A lower failure rate indicates a more reliable engine.

🔍 Click to enlarge

Figure 4: Share of cases with no confident device name (failure rate).

Fingerbank recorded the lowest failure rate at just 3% (7 of 206 cases), narrowly ahead of DeepSeek v4 Pro at 6%. DeepSeek v4 Flash followed at 11%. The Kimi models exhibited markedly higher failure rates—19% for k2.5 and 32% for k2.6—indicating a much greater propensity to return unidentifiable results, with nearly one in three cases left unresolved by Kimi k2.6.

The benchmark results, now spanning DeepSeek and Kimi open-weight models, reinforce our consistent finding: For network device fingerprinting, a specialized engine like Fingerbank remains vastly superior to general-purpose LLMs.

While the newer DeepSeek and Kimi models show varying strengths—DeepSeek v4 Flash in cost—none can match Fingerbank’s combined performance in the critical areas of real-time latency, cost efficiency, and, most importantly, accuracy across diverse device categories and signal combinations. Fingerbank offers:

Real-time Performance: Sub-second response times (488 ms) essential for inline network operations, against LLM latencies measured in tens of seconds to minutes.

Cost Efficiency: Flat, predictable pricing at a fraction of the cost—up to 30× cheaper than the priciest LLM, with none of the per-scenario cost volatility.

Superior Accuracy: Consistently higher accuracy (85%) and manufacturer match rate (91%), avoiding the confident hallucinations and high failure rates observed in LLMs.

As AI continues to evolve, there may be roles for LLMs in offline analysis or anomaly detection. However, for inline, real-time device fingerprinting, Fingerbank’s purpose-built architecture remains the undisputed champion.

View full diagnostic results