| model | payloads | runs | hallucination % | matched/total | mean score | mean latency | mean cost |
|---|---|---|---|---|---|---|---|
| DeepSeek-v4-Flash | 10 | 10 | 16% | 84/100 | 79 | 10214 ms | $0.00043 |
| Kimi-k2.5 | 10 | 10 | 17% | 71/84 | 73 | 83854 ms | $0.00602 |
| Kimi-k2.6 | 10 | 10 | 18% | 57/67 | 55 | 159945 ms | $0.01246 |
| DeepSeek-v4-Pro | 10 | 10 | 23% | 77/100 | 81 | 15408 ms | $0.00166 |
Payload: {'oui': '0003ac', 'dhcp_fingerprint': '1,3,6,12,15,28,42', 'dhcp_vendor': 'udhcp 1.22.1'} · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-v4-Flash | ≈1 Router, Access Point or Femtocell/Embedded Linux (60) Unknown | ≈2 Router, Access Point or Femtocell/Digi International (65) Digi International Inc. | ≈3 Printer or Scanner/Hewlett Packard (70) Hewlett Packard | ≈4 Router, Access Point or Femtocell/SonicWALL (70) SonicWALL, Inc. | ≈5 VoIP Device/Digium (65) Digium, Inc. | ≈3 Printer or Scanner/HP (70) Hewlett-Packard | ≈6 Router, Access Point or Femtocell/ZyXEL (75) ZyXEL Communications Corp. | ≈7 Router, Access Point or Femtocell/Hewlett-Packard/Unknown (40) Hewlett-Packard Company | ≈8 Router, Access Point or Femtocell/AVM/FRITZ!Box (85) AVM GmbH | ≈8 Router, Access Point or Femtocell/AVM (70) AVM GmbH | ✗ 80% |
| DeepSeek-v4-Pro | ≈1 Audio, Imaging or Video Equipment/Axis Communications/IP Camera (85) Axis Communications AB | ≈2 IoT (Internet of Things)/IBM/Generic Embedded Device (60) IBM Corp | ≈3 Operating System/Linux (40) FIBERON Technologies, Inc. | ≈4 Router, Access Point or Femtocell/Cisco (60) Cisco Systems, Inc. | ≈3 Operating System/Linux (70) Aware, Inc. | ≈5 Audio, Imaging or Video Equipment/Sonos (80) Sonos, Inc. | ≈2 IoT (Internet of Things)/Pixel System/Embedded Device (60) Pixel System, Inc. | ≈6 Audio, Imaging or Video Equipment/Sony/Smart TV (85) Sony Corporation | ≈4 Router, Access Point or Femtocell/Generic (35) M-Systems Flash Disk Pioneers, Ltd. | ≈7 Router, Access Point or Femtocell/FiberLogic Communications (60) FiberLogic Communications Inc. | ✗ 80% |
| Kimi-k2.5 | ≈1 Router, Access Point or Femtocell/Atheros Communications (55) Atheros Communications, Inc. | ≈1 Router, Access Point or Femtocell/Atheros Communications (60) Atheros Communications, Inc. | ERROR (ReadTimeout) (0) | ≈1 Router, Access Point or Femtocell/Atheros (65) Atheros Communications, Inc. | ERROR (ReadTimeout) (0) | ≈3 Router, Access Point or Femtocell/Accton Technology Corporation (65) Accton Technology Corporation | ≈4 Router, Access Point or Femtocell/Apple/AirPort (85) Apple, Inc. | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ✗ 40% |
| Kimi-k2.6 | ERROR (ReadTimeout) (0) | ≈2 Operating System/Linux (65) | ≈2 Operating System/Linux (60) Unknown | ≈3 Router, Access Point or Femtocell/Atheros Communications, Inc. (60) Atheros Communications, Inc. | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ≈3 Router, Access Point or Femtocell/Atheros Communications, Inc. (65) Atheros Communications, Inc. | ERROR (ReadTimeout) (0) | ≈2 Operating System/Linux (65) | ERROR (ReadTimeout) (0) | ✗ 40% |
Payload: {'oui': '001070', 'dhcp_fingerprint': '1,3,6,15,42,44,46,47', 'hostname': 'TREND_18_2B_19'} · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 Router, Access Point or Femtocell/TRENDnet (70) TRENDnet | ≈1 Router, Access Point or Femtocell/Trend (85) Trend | ≈2 Audio, Imaging or Video Equipment/Camera/TRENDnet (70) TRENDnet, Inc. | ≈1 Router, Access Point or Femtocell/Trendnet (70) Trendnet, Inc. | ≈1 Router, Access Point or Femtocell/TRENDnet (60) TRENDnet | ≈1 Router, Access Point or Femtocell/TRENDnet (70) TRENDnet, Inc. | ≈1 Router, Access Point or Femtocell/TRENDnet (75) Trendware, Inc. | ≈1 Router, Access Point or Femtocell/Trendnet/Unknown Model (60) Trendnet | ≈1 Router, Access Point or Femtocell/Trendnet (85) Trendware International Corp. | ≈1 Router, Access Point or Femtocell/TRENDnet (70) TRENDnet, Inc. | ✗ 10% |
| DeepSeek-v4-Pro | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (70) TRENDnet, Inc. | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (75) TRENDware International, Inc. | ≈2 Router, Access Point or Femtocell/TRENDnet/Unknown (60) TRENDware International, Inc. | ≈2 Router, Access Point or Femtocell/TRENDnet/TRENDnet Device (70) TRENDware International, Inc. | ≈3 Router, Access Point or Femtocell/ZyXEL (65) ZyXEL Communications Corp. | ≈2 Router, Access Point or Femtocell/TRENDnet/TRENDware (92) TRENDware International, Inc. | ≈2 Router, Access Point or Femtocell/TRENDnet/TRENDnet Device (55) TRENDnet | ≈4 Switch and Wireless Controller/TRENDnet/Switch (85) TRENDnet | ≈2 Router, Access Point or Femtocell/TRENDnet/Unknown (85) TRENDnet | ≈2 Router, Access Point or Femtocell/TRENDnet (65) TRENDnet | ✗ 40% |
| Kimi-k2.5 | ≈1 Storage Device/TRENDnet (85) TRENDnet (TRENDware International Inc.) | ≈2 Router, Access Point or Femtocell/TrendChip Technology Corp. (75) TrendChip Technology Corp. | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (75) TRENDnet, Inc. | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (70) TRENDware International Inc. | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (75) TRENDnet | ≈3 Audio, Imaging or Video Equipment/TRENDnet (65) TRENDnet | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (85) TRENDnet | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (70) TRENDnet | ≈3 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (85) TRENDnet | ERROR (ReadTimeout) (0) | ✗ 22% |
| Kimi-k2.6 | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (75) TRENDnet, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (70) TRENDware International, Inc. | ERROR (ReadTimeout) (0) | ≈3 IoT (Internet of Things)/TRENDnet (65) TRENDware International Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (65) TRENDnet, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/TRENDnet/IP Camera (65) TRENDnet | ≈1 Audio, Imaging or Video Equipment/TRENDnet (70) TRENDnet, Inc. | ✗ 17% |
Payload: {'oui': '001d63', 'dhcp_fingerprint': '1,3,28,6', 'hostname': 'Miele-001D63FFFE754702', 'mdns_services': ['_mieleatho… · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 IoT/Miele/Miele@home Appliance (90) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Home Appliance (90) Miele & Cie. KG | ≈1 IoT/Miele/AtHome (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele (95) Miele & Cie. KG | ≈1 IoT/Home Appliance/Miele (90) Miele & Cie. KG | ≈1 IoT/Appliance/Miele (90) Miele & Cie. KG | ≈1 IoT/Miele/Miele Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Home Appliance (90) Miele & Cie. KG | ≈1 IoT/Miele/Home Appliance (95) Miele & Cie. KG | ≈1 IoT/Miele/Miele@home Appliance (85) Miele & Cie. KG | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 IoT/Miele Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Unknown Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Miele@home Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Miele@home Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Miele@home Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Home Automation/Miele (95) Miele & Cie. KG | ≈1 IoT/Miele Appliance (95) Miele & Cie. KG | ≈1 IoT/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/@home Device (95) Miele & Cie. KG | ≈1 IoT/Miele/Miele@home Appliance (98) Miele & Cie. KG | ✓ 0% |
| Kimi-k2.5 | ≈1 IoT (Internet of Things)/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Connected Appliance (95) Miele & Cie. KG | ≈1 IoT/Miele (95) Miele & Cie. KG | ≈1 IoT/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele (95) Miele & Cie. KG | ≈1 IoT/Miele (95) Miele & Cie. KG | ≈1 IoT/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Smart Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Smart Appliance (95) Miele & Cie. KG | ✓ 0% |
| Kimi-k2.6 | ≈1 IoT (Internet of Things)/Home Appliance/Miele (95) Miele & Cie. KG | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Miele (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Miele@home (98) Miele & Cie. KG | ≈1 IoT/Miele/Smart Appliance (95) Miele | ≈1 IoT (Internet of Things)/Miele/Smart Appliance (98) Miele & Cie. KG | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Miele/Miele@home (95) Miele & Cie. KG | ≈1 IoT/Miele/Miele@home Appliance (95) Miele & Cie. KG | ≈1 IoT (Internet of Things)/Miele/Home Appliance (95) Miele & Cie. KG | ✓ 0% |
Payload: {'oui': '38420b', 'upnp_server_strings': ['Linux UPnP/1.0 Sonos/94.1-75110 (ZPS31)', 'Linux UPnP/1.0 Sonos/94.1-75110… · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Wireless Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos System (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (90) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Play:1 (85) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (85) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (90) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos One (Gen 2) (80) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (80) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ✓ 0% |
| Kimi-k2.5 | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos One (95) Sonos, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos One (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Smart Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Move (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Smart Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Sonos One (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ✓ 0% |
| Kimi-k2.6 | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (90) Sonos, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/Sonos/Speaker (95) Sonos, Inc. | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/Sonos (90) Sonos, Inc. | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/Sonos (95) Sonos, Inc. | ERROR (ReadTimeout) (0) | ✓ 0% |
Payload: {'oui': '48e7da', 'hostname': 'EcoNet-48E7DAF88C48', 'user_agents': ['NX NIFM/00'], 'tcp_syn_signatures': ['4:255+0:0… · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 Router, Access Point or Femtocell/EcoNet (75) EcoNet (Shanghai) Co., Ltd. | ≈1 Router, Access Point or Femtocell/EcoNet (60) EcoNet (Shanghai) Limited | ≈1 Router, Access Point or Femtocell/EcoNet (85) EcoNet (Shanghai) Co., Ltd. | ≈1 Router, Access Point or Femtocell/EcoNet (70) EcoNet | ≈1 Router, Access Point or Femtocell/EcoNet/EcoNet Router (70) EcoNet (Shanghai) Co., Ltd. | ≈2 IoT (Internet of Things)/EcoNet (70) EcoNet (Shanghai) Co., Ltd. | ≈3 Audio, Imaging or Video Equipment/IP Camera (65) Chicony Electronics Co., Ltd. | ≈1 Router, Access Point or Femtocell/EcoNet/EcoNet Device (70) EcoNet (Shanghai) Co., Ltd. | ≈3 Audio, Imaging or Video Equipment/IP Camera (70) AzureWave Technology Inc. | ≈2 IoT (Internet of Things)/EcoNet (65) EcoNet (Shanghai) Limited | ✗ 40% |
| DeepSeek-v4-Pro | ≈1 IoT/Smart Home/Water Heater/Rheem EcoNet (85) Rheem Manufacturing Company | ≈2 Audio, Imaging or Video Equipment/Hikvision/Network Camera (85) Hangzhou Hikvision Digital Technology Co | ≈3 Router, Access Point or Femtocell/Eero/Eero (92) Eero Inc. | ≈1 IoT (Internet of Things)/Econet/Econet Smart Device (80) Econet (Robotics) Pte. Ltd. | ≈4 Router, Access Point or Femtocell/EcoNet/EcoNet Router (90) EcoNet (Shenzhen) Co., Ltd. | ≈1 IoT (Internet of Things)/EcoNet Controls/EcoNet Device (92) EcoNet Controls, Inc. | ≈1 IoT/EcoNet/Smart Hub (85) EcoNet | ≈4 Router, Access Point or Femtocell/Econet/EcoNet Router (85) Econet (Pvt) Ltd. | ≈1 IoT (Internet of Things)/EcoNet (65) EcoNet (Shenzhen) Technology Co., Ltd. | ≈5 Audio, Imaging or Video Equipment/TP-Link/IP Camera (85) TP-LINK TECHNOLOGIES CO., LTD. | ✗ 50% |
| Kimi-k2.5 | ≈1 Gaming Console/Nintendo/Nintendo Switch (92) Nintendo Co., Ltd. | ≈2 Automotive, Energy and Tools/Rheem/EcoNet (90) Rheem Manufacturing Company | ≈1 Gaming Console/Nintendo/Switch (90) Nintendo Co., Ltd. | ≈2 Automotive, Energy and Tools/Rheem/EcoNet Smart Water Heater (95) Rheem Manufacturing Company | ERROR (ReadTimeout) (0) | ≈2 Automotive, Energy and Tools/Rheem/EcoNet Controller (95) Rheem Manufacturing Company | ≈2 Automotive, Energy and Tools/Rheem/EcoNet (90) Rheem Manufacturing Company | ERROR (ReadTimeout) (0) | ≈2 Automotive, Energy and Tools/Rheem/EcoNet Smart Controller (92) Rheem Manufacturing Company | ≈2 Automotive, Energy and Tools/Rheem/EcoNet (95) Rheem Manufacturing Company | ✗ 25% |
| Kimi-k2.6 | ≈1 Automotive, Energy and Tools/Rheem/EcoNet (62) Rheem Manufacturing Company | ERROR (ReadTimeout) (0) | ≈3 Gaming Console/Nintendo/Nintendo Switch (95) Nintendo Co., Ltd. | ≈1 Automotive, Energy and Tools/Rheem/EcoNet (85) Rheem Manufacturing Company | ≈3 Gaming Console/Nintendo/Switch (85) Nintendo Co., Ltd. | ERROR (ReadTimeout) (0) | ≈1 Automotive, Energy and Tools/EcoNet (60) EcoNet Controls | ≈1 Automotive, Energy and Tools/Energy Management/EcoNet Controller (85) Rheem Manufacturing Company | ≈3 Gaming Console/Nintendo/Nintendo Switch (90) Nintendo Co., Ltd. | ≈1 Automotive, Energy and Tools/Rheem/EcoNet (85) Rheem Manufacturing Company | ✗ 38% |
Payload: {'oui': '98ed5c', 'dhcp_fingerprint': '1,3,28,6', 'user_agents': ['WMSDK']} · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 Gaming Console/Microsoft/Xbox (70) Microsoft Corporation | ≈1 Gaming Console/Microsoft/Xbox 360 (70) Microsoft Corp | ≈2 Audio, Imaging or Video Equipment/Media Player (30) AzureWave Technology Inc. | ≈2 Audio, Imaging or Video Equipment/Media Player (40) AzureWave Technology Inc. | ≈3 Gaming Console/Nintendo/Wii (50) Nintendo Co., Ltd. | ≈2 Audio, Imaging or Video Equipment/ViewSonic/Media Player (50) Shenzhen ViewSonic Technology Co., Ltd. | ≈2 Audio, Imaging or Video Equipment/Microsoft/Windows Media Extender (55) Microsoft Corporation | ≈2 Audio, Imaging or Video Equipment/Samsung/Media Player (55) Samsung Electronics Co., Ltd | ≈2 Audio, Imaging or Video Equipment/Unknown/Media Player (40) | ≈2 Audio, Imaging or Video Equipment/Samsung/Smart TV (60) Samsung Electronics Co., Ltd | ✗ 30% |
| DeepSeek-v4-Pro | ≈1 Automotive, Energy and Tools/Tesla/Tesla Vehicle (80) Tesla, Inc. | ≈1 Automotive, Energy and Tools/Tesla (80) Tesla, Inc. | ≈2 Audio, Imaging or Video Equipment/TP-Link/IP Camera (45) TP-Link Technologies Co., Ltd. | ≈1 Automotive, Energy and Tools/Tesla (80) Tesla, Inc. | ≈3 Tablet/Amazon/Kindle (75) Amazon Technologies Inc. | ≈4 IoT (Internet of Things)/Realtek Semiconductor Corp./Generic IoT Device (60) Realtek Semiconductor Corp. | ≈5 IoT (Internet of Things)/Google/Home (60) Google, Inc. | ≈6 IoT (Internet of Things) (60) Sichuan Tianyi Digital Home Network Tech | ≈1 Automotive, Energy and Tools/Tesla/Vehicle (65) Tesla, Inc. | ≈7 Audio, Imaging or Video Equipment/Linkplay Technology/WiiM Mini (90) Linkplay Technology Inc. | ✗ 60% |
| Kimi-k2.5 | ≈1 Audio, Imaging or Video Equipment/Skyworth (70) Shenzhen Skyworth Digital Technology Co. | ≈2 Gaming Console/Microsoft/Xbox (75) Microsoft Corporation | ≈3 Audio, Imaging or Video Equipment/Media Streaming Device (55) Unknown | ERROR (ReadTimeout) (0) | ≈5 Router, Access Point or Femtocell/Aruba Networks/Access Point (75) Aruba Networks | ERROR (ReadTimeout) (0) | ≈1 Audio, Imaging or Video Equipment/Skyworth/Smart TV (75) Skyworth Digital Technology Co., Ltd. | ≈6 Audio, Imaging or Video Equipment/Apple (75) Apple, Inc. | ≈6 Audio, Imaging or Video Equipment/Apple/Apple TV (65) Apple, Inc. | ≈7 Laptop or Desktop/Lenovo (70) Lenovo | ✗ 75% |
| Kimi-k2.6 | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ERROR (ReadTimeout) (0) | ≈2 IoT (Internet of Things)/Shanghai High-Flying Electronics (50) Shanghai High-Flying Electronics Technol | ERROR (ReadTimeout) (0) | ≈3 Audio, Imaging or Video Equipment/Media Player (45) | ERROR (ReadTimeout) (0) | ≈4 Operating System/Microsoft/Windows (45) | ERROR (ReadTimeout) (0) | ✗ 67% |
Payload: {'oui': 'b827eb', 'hostname': '365id-0000021243702441'} · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 Generic PC/Raspberry Pi (80) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (60) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi (80) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi (50) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi (70) Raspberry Pi Foundation | ≈1 IoT/Raspberry Pi Foundation (60) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (70) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (60) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (80) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (80) Raspberry Pi Foundation | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 Generic PC/Raspberry Pi (40) Raspberry Pi Foundation | ≈1 IoT/Raspberry Pi Foundation/365id (80) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi Foundation/365id Device (35) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/365id (70) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (35) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (35) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi Foundation/365id Device (60) Raspberry Pi Foundation | ≈1 IoT/Raspberry Pi/365id (60) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (65) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi Foundation/Unknown Model (65) Raspberry Pi Foundation | ✓ 0% |
| Kimi-k2.5 | ≈1 Generic PC/Raspberry Pi (90) Raspberry Pi (Trading) Ltd. | ≈2 IoT (Internet of Things)/365id/ID Verification Terminal (70) Raspberry Pi Foundation | ≈1 IoT/Raspberry Pi/Raspberry Pi (70) Raspberry Pi (Trading) | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (65) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi/Raspberry Pi (90) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (90) Raspberry Pi Foundation | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Raspberry Pi/365id (80) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (70) Raspberry Pi Foundation | ERROR (ReadTimeout) (0) | ✗ 12% |
| Kimi-k2.6 | ≈1 IoT (Internet of Things)/Raspberry Pi Foundation/Raspberry Pi (65) Raspberry Pi Foundation | ≈1 IoT (Internet of Things)/Raspberry Pi Foundation/Raspberry Pi (65) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (70) Raspberry Pi Foundation | ERROR (ReadTimeout) (0) | ≈3 IoT/365id (70) 365id | ≈4 Printer or Scanner/Scanner/ID Scanner (70) 365id | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (70) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi Foundation/Unknown (40) Raspberry Pi Foundation | ≈1 Generic PC/Raspberry Pi (Trading) Ltd/Raspberry Pi (85) Raspberry Pi (Trading) Ltd | ≈1 Generic PC/Raspberry Pi Foundation/Raspberry Pi (85) Raspberry Pi Foundation | ✗ 22% |
Payload: {'oui': 'dc0675', 'dhcp_fingerprint': '1,3,28,6', 'hostname': 'Smart Cat Litter Box C20', 'tcp_syn_signatures': ['4:6… · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 IoT/Smart Cat Litter Box C20 (90) Shenzhen Hejikeji Technology Co., Ltd. | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (95) Unknown | ≈1 IoT/Other/Smart Cat Litter Box C20 (95) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box (80) Shenzhen Herun Intelligence Technology C | ≈1 IoT/Other/Smart Cat Litter Box C20 (85) Unknown | ≈1 IoT/Smart Home/Smart Cat Litter Box C20 (80) Unknown | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (95) | ≈1 IoT/Smart Cat Litter Box C20 (85) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (90) Unknown | ≈1 IoT/Smart Litter Box (90) | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 IoT (Internet of Things)/Shenzhen Zoolife Tech Co., Ltd./Smart Cat Litter Box C20 (95) Shenzhen Zoolife Tech Co., Ltd. | ≈1 IoT/Smart Cat Litter Box C20 (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Sichuan AI-Link Technology/Smart Cat Litter Box C20 (95) Sichuan AI-Link Technology Co., Ltd. | ≈1 IoT/Smart Cat Litter Box C20 (85) August Home Inc. | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Smart Pet Care/Smart Cat Litter Box C20 (95) Espressif Inc. | ≈1 IoT/Smart Cat Litter Box C20 (95) Shenzhen Concox Information Technology C | ✓ 0% |
| Kimi-k2.5 | ≈1 IoT (Internet of Things)/Pet Care/Smart Cat Litter Box (95) Smart Pet Care (Hangzhou) Technology Co. | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Pet Care/Smart Cat Litter Box/C20 (98) Unknown | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box/C20 (98) Unknown | ≈1 IoT/Smart Cat Litter Box (95) | ≈1 IoT (Internet of Things)/PETKIT/Smart Cat Litter Box C20 (95) PETKIT Innovation Technology (Shenzhen) | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Smart Home/Smart Pet Device/Smart Cat Litter Box (95) Unknown | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (95) Eresson Technology Co., Ltd. | ✓ 0% |
| Kimi-k2.6 | ≈1 IoT/Smart Home/Pet Care/Smart Cat Litter Box C20 (90) Unknown | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (90) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (80) | ≈1 IoT/Smart Cat Litter Box C20 (95) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box/C20 (95) | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box/C20 (95) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box C20 (85) | ≈1 IoT (Internet of Things)/Smart Cat Litter Box (85) | ERROR (ReadTimeout) (0) | ✓ 0% |
Payload: {'oui': 'e80690', 'mdns_services': ['_esphomelib._tcp.local'], 'tcp_syn_signatures': ['4:64+0:0:1440:mss*4,0:mss::0']} · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 IoT/Espressif/ESP32 (90) Espressif Inc. | ≈1 IoT/Espressif/ESP32 (90) Espressif Inc. | ≈1 IoT/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT/Espressif/ESPHome (90) Espressif Inc. | ≈1 IoT/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT/ESPHome (85) Espressif Systems | ≈1 IoT/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT/ESPHome (90) Espressif Inc. | ≈1 IoT/Espressif/ESP Home Device (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome (90) Espressif Inc. | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 IoT (Internet of Things)/Espressif/ESPHome Device (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif Systems/ESPHome Device (95) Espressif Systems | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Systems | ≈1 IoT (Internet of Things)/Espressif/ESP32 (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome Device (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Other/ESPHome Device (90) Samsung Electro-Mechanics(Thailand) Co., | ≈1 IoT/Espressif/ESPHome Device (85) Espressif Inc. | ≈1 IoT/Espressif/ESP32 (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome Device (90) Espressif Systems, Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome (90) Espressif Inc. | ✓ 0% |
| Kimi-k2.5 | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Home Automation/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif Inc./ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/ESPHome (95) Espressif Systems (Shanghai) Co., Ltd. | ≈1 IoT (Internet of Things)/Home Automation/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Home Automation/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/ESPHome (95) Espressif Systems (Shanghai) Co., Ltd. | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome Device (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ✓ 0% |
| Kimi-k2.6 | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif Systems/ESPHome (95) Espressif Systems (Shanghai) Co., Ltd. | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Espressif Inc./ESPHome (98) Espressif Inc. | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Espressif Systems (95) Espressif Systems | ≈1 IoT (Internet of Things)/Espressif/ESPHome (95) Espressif Inc. | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/ESPHome (95) Espressif Inc. | ✓ 0% |
Payload: {'oui': 'e868e7', 'dhcp_fingerprint': '1,3,28,6,42', 'user_agents': ['shelly-plug-s/1.0 (MongooseOS/202301031927; esp… · 10 run(s) | |||||||||||
| model | run 1 | run 2 | run 3 | run 4 | run 5 | run 6 | run 7 | run 8 | run 9 | run 10 | hallucination |
| DeepSeek-v4-Flash | ≈1 IoT/Allterco/Shelly Plug S (95) Allterco Robotics | ≈1 IoT/Smart Plug/Shelly Plug S (95) Shelly Group Ltd | ≈1 IoT/Allterco Robotics/Shelly Plug S (95) Allterco Robotics | ≈1 IoT/Smart Plug/Shelly Plug S (95) Shelly (Allterco Robotics) | ≈1 Automotive, Energy and Tools/Energy/Shelly Plug S (95) Allterco Robotics Ltd. | ≈1 IoT (Internet of Things)/Allterco/Shelly Plug S (95) Allterco Robotics Ltd. | ≈1 IoT/Smart Plug/Shelly Plug S (95) Allterco Robotics | ≈1 IoT (Internet of Things)/Smart Plug/Shelly Plug S (95) Shelly Group Ltd. | ≈1 IoT/Smart Plug/Shelly Plug S (95) Shelly Group | ≈1 IoT/Shelly/Plug S (95) Shelly | ✓ 0% |
| DeepSeek-v4-Pro | ≈1 IoT (Internet of Things)/Smart Plug/Shelly Plug S (90) Espressif Inc. | ≈1 IoT (Internet of Things)/Shelly Plug S (95) Allterco Robotics | ≈1 IoT (Internet of Things)/Smart Plug/Shelly Plug S (95) Espressif Inc. | ≈1 IoT (Internet of Things)/Shelly/Plug S (100) Allterco Robotics | ≈1 IoT (Internet of Things)/Shelly/Plug S (95) Shelly | ≈1 IoT/Shelly/Plug S (95) Shelly | ≈1 IoT (Internet of Things)/Allterco Robotics/Shelly Plug S (100) Allterco Robotics | ≈1 IoT (Internet of Things)/Shelly/Shelly Plug S (100) Shelly | ≈1 IoT/Shelly/Shelly Plug S (95) Allterco Robotics | ≈1 IoT/Shelly/Shelly Plug S (98) Shelly | ✓ 0% |
| Kimi-k2.5 | ≈1 Automotive, Energy and Tools/Shelly/Plug S (100) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Shelly/Plug S (95) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (100) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (98) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (95) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Shelly/Shelly Plug S (100) Allterco Robotics | ≈1 Automotive, Energy and Tools/Shelly/Plug S (98) Shelly Group (Allterco Robotics) | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (95) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (100) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Shelly/Plug S (95) Allterco Robotics | ✓ 0% |
| Kimi-k2.6 | ≈1 IoT (Internet of Things)/Shelly/Shelly Plug S (100) Allterco Robotics Ltd. | ERROR (ReadTimeout) (0) | ≈1 IoT (Internet of Things)/Shelly/Plug S (100) Shelly | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (100) Allterco Robotics Ltd. | ≈1 Automotive, Energy and Tools/Allterco Robotics/Shelly Plug S (100) Allterco Robotics | ERROR (ReadTimeout) (0) | ≈1 Automotive, Energy and Tools/Shelly/Shelly Plug S (95) Shelly | ≈1 Automotive, Energy and Tools/Shelly/Plug S (99) Shelly | ERROR (ReadTimeout) (0) | ≈1 Automotive, Energy and Tools/Shelly/Plug S (95) Shelly | ✓ 0% |