[Q3-2026] Measuring LLM Consistency: A Hallucination Benchmark for Device Identification Part 2
Fingerbank Team
Executive Summary
This benchmark evaluates a simple but critical property of Large Language Models (LLMs): consistency. In this study, four leading Chinese models each received the same payload 10 times. A reliable model should produce the same device identification repeatedly; any variability across repeated runs is treated as a form of hallucination.
DeepSeek-v4-Flash achieved the strongest consistency in this cohort with a 16% hallucination rate, while also being the fastest and cheapest model tested. In contrast, DeepSeek-v4-Pro showed the highest instability, with a 23% hallucination rate — meaning the larger “Pro” model was the least deterministic despite earning the highest mean quality score.
To ensure a rigorous comparison, we defined hallucination as instability. If the same input leads to different device names or manufacturers across runs, the model is considered to be hallucinating.
Payloads: 10 unique device-identification prompts.
Repeated Runs: Each payload was submitted 10 times to each model (100 total runs per model).
Hallucination Metric: For each payload, we identified the most common answer. The hallucination rate is computed as:
1 − (most-common-answer frequency ÷ total runs) This value is then averaged across all payloads.
Answer Matching: Two answers are treated as equivalent if:
Device names are fuzzily similar (≈0.7) AND manufacturers match.
The manufacturer, category, and vendor (first two segments of the device-name path) all match.
The following table summarizes the performance of the 4 models compared in this benchmark. While consistency is our primary metric, we also tracked latency and cost to provide a complete picture of operational efficiency. Note that the Matched / Total column reveals a second story: both Kimi models failed to return a usable answer on a number of runs (timeouts and errors), shrinking their valid sample to 84 and 67 runs respectively.
Based on the data, we can categorize the models into specific performance profiles:
🏆 Best Stability & Efficiency: DeepSeek-v4-Flash DeepSeek-v4-Flash delivered the lowest hallucination rate (16%), the lowest cost ($0.00043), and strong latency (10,214 ms). It is the clear all-round choice for production systems that rely on deterministic enrichment and classification.
⚖️ Highest Quality, Lower Reliability: DeepSeek-v4-Pro DeepSeek-v4-Pro earned the highest mean score (81) but also the highest hallucination rate (23%). It tends to produce richer answers, yet varies the most across identical prompts — a trade-off of quality for determinism.
⚠️ Operationally Expensive: Kimi-k2.6 Kimi-k2.6 was by far the slowest (159,945 ms — roughly 2.7 minutes per run on average) and most expensive ($0.01246), while also recording the lowest mean score (55) and the most failed runs (only 67 of 100 returned).
🕒 Middle Ground with Caveats: Kimi-k2.5 Kimi-k2.5 posted a competitive 17% hallucination rate, but its high latency (83,854 ms) and 16 missing runs make it harder to recommend for latency-sensitive or high-throughput workloads.
Per-model view: the chart above plots, for a single model, how many distinct devices it produced for each payload. A value of 1 means perfect consistency; higher values indicate the model returned different identities for the same input. DeepSeek-v4-Flash diverged most sharply on payload 1 (8 distinct devices) but held to a single answer on the majority of payloads.
In device intelligence systems like Fingerbank, consistency is often as important as accuracy. If the same network signature is mapped to different device identities on different runs, downstream systems — such as security policies, network analytics, and inventory management — become unreliable.
A low hallucination rate indicates that a model is not only capable of producing correct answers but is also deterministic enough to produce them repeatedly. This cohort also surfaces a related operational risk: a model that times out or errors on a fraction of runs (as the Kimi models did) introduces gaps that are just as damaging to an automated pipeline as an inconsistent answer.
The benchmark dashboard allows us to inspect individual runs. The comparison below overlays all four models, making it easy to see which payloads triggered divergence. Note the similarity clusters (marked ≈1, ≈2) which highlight where models agreed or diverged.
Determinism varies across models: hallucination rates ranged from 16% to 23% under identical conditions, even within this single-region cohort.
Bigger is not always more stable: DeepSeek-v4-Pro scored highest on quality (81) yet hallucinated the most (23%), while the lighter Flash variant was the most consistent.
Reliability includes availability: the Kimi models lost 16–33 runs to timeouts and errors, a failure mode that traditional accuracy tests would not capture.
Cost and latency diverge widely: Kimi-k2.6 was roughly 29× more expensive and 15× slower than DeepSeek-v4-Flash, with no consistency benefit to show for it.
Benchmarking repeated prompts is valuable: instability and intermittent failures only become visible when the same query is executed many times.
This analysis was generated from a specialized dashboard built to compare LLM behavior across repeated identical prompts. The benchmark focused specifically on device identification tasks, measuring how consistently each model returned a normalized device identity.
For more insights into device intelligence and AI-driven classification, visit the Fingerbank Blog.