On the Origin of Scanning: The Impact of Location on Internet-Wide Scans
Fast IPv4 scanning has enabled researchers to answer a wealth of security and networking questions. Yet, despite widespread use, there has been little validation of the methodology’s accuracy, including whether a single scan provides sufficient coverage. In this paper, we analyze how scan origin affects the results of Internet-wide scans by completing three HTTP, HTTPS, and SSH scans from seven geographically and topologically diverse networks. We find that individual origins miss an average 1.6–8.4% of HTTP, 1.5–4.6% of HTTPS, and 8.3–18.2% of SSH hosts. We analyze why origins see different hosts, and show how permanent and temporary blocking, packet loss, geographic biases, and transient outages affect scan results. We discuss the implications for scanning and provide recommendations for future studies.
Study Details
- Study
-
On the Origin of Scanning: The Impact of Location on Internet-Wide Scans
ACM Internet Measurement Conference (IMC), October 2020 - Authors
- Gerry Wan, Liz Izhikevich, David Adrian, Katsunari Yoshioka, Ralph Holz, Christian Rossow, Zakir Durumeric
- Contact
- Gerry Wan
Dataset Details
We provide access to the raw ZMap and ZGrab scans that are run from each network perspectives across three trials.
File Download
File Name | MetaData | SHA-1 Fingerprint | Size | Updated At |
---|---|---|---|---|
trial1_dns_australia_zmap.json.lz4 | unavailable | unavailable | 22.8 GB | 2020-09-15 |
trial1_dns_brazil_zmap.json.lz4 | unavailable | unavailable | 23.33 GB | 2020-09-15 |
trial1_dns_censys_zmap.json.lz4 | unavailable | unavailable | 22.6 GB | 2020-09-15 |
trial1_dns_germany_zmap.json.lz4 | unavailable | unavailable | 23.24 GB | 2020-09-15 |
trial1_dns_japan_zmap.json.lz4 | unavailable | unavailable | 22.72 GB | 2020-09-15 |
trial1_dns_stanford1_zmap.json.lz4 | unavailable | unavailable | 22.99 GB | 2020-09-15 |
trial1_dns_stanford64_zmap.json.lz4 | unavailable | unavailable | 30.09 GB | 2020-09-15 |
trial1_http_australia_zgrab.json.lz4 | unavailable | unavailable | 155.95 GB | 2019-10-25 |
trial1_http_australia_zmap.json.lz4 | unavailable | unavailable | 13.74 GB | 2020-09-15 |
trial1_http_brazil_zgrab.json.lz4 | unavailable | unavailable | 157.01 GB | 2019-10-25 |
trial1_http_brazil_zmap.json.lz4 | unavailable | unavailable | 14.21 GB | 2020-09-15 |
trial1_http_censys_zgrab.json.lz4 | unavailable | unavailable | 150.7 GB | 2019-10-24 |
trial1_http_censys_zmap.json.lz4 | unavailable | unavailable | 13.22 GB | 2020-09-15 |
trial1_http_germany_zgrab.json.lz4 | unavailable | unavailable | 161.42 GB | 2019-10-24 |
trial1_http_germany_zmap.json.lz4 | unavailable | unavailable | 13.6 GB | 2020-09-15 |
trial1_http_japan_zgrab.json.lz4 | unavailable | unavailable | 158.56 GB | 2019-10-24 |
trial1_http_japan_zmap.json.lz4 | unavailable | unavailable | 14.0 GB | 2020-09-15 |
trial1_http_stanford1_zgrab.json.lz4 | unavailable | unavailable | 158.25 GB | 2019-10-24 |
trial1_http_stanford1_zmap.json.lz4 | unavailable | unavailable | 13.79 GB | 2020-09-15 |
trial1_http_stanford64_zgrab.json.lz4 | unavailable | unavailable | 158.49 GB | 2019-10-24 |
trial1_http_stanford64_zmap.json.lz4 | unavailable | unavailable | 18.88 GB | 2020-09-15 |
trial1_ssh_australia_zgrab.json.lz4 | unavailable | unavailable | 594.47 MB | 2019-10-26 |
trial1_ssh_australia_zmap.json.lz4 | unavailable | unavailable | 11.29 GB | 2020-09-15 |
trial1_ssh_brazil_zgrab.json.lz4 | unavailable | unavailable | 602.51 MB | 2019-10-26 |
trial1_ssh_brazil_zmap.json.lz4 | unavailable | unavailable | 12.79 GB | 2020-09-15 |
trial1_ssh_censys_zgrab.json.lz4 | unavailable | unavailable | 575.88 MB | 2019-10-26 |
trial1_ssh_censys_zmap.json.lz4 | unavailable | unavailable | 11.46 GB | 2020-09-15 |
trial1_ssh_germany_zgrab.json.lz4 | unavailable | unavailable | 647.61 MB | 2019-10-26 |
trial1_ssh_germany_zmap.json.lz4 | unavailable | unavailable | 11.61 GB | 2020-09-15 |
trial1_ssh_japan_zgrab.json.lz4 | unavailable | unavailable | 607.1 MB | 2019-10-26 |
trial1_ssh_japan_zmap.json.lz4 | unavailable | unavailable | 12.49 GB | 2020-09-15 |
trial1_ssh_stanford1_zgrab.json.lz4 | unavailable | unavailable | 586.43 MB | 2019-10-26 |
trial1_ssh_stanford1_zmap.json.lz4 | unavailable | unavailable | 12.48 GB | 2020-09-15 |
trial1_ssh_stanford64_zgrab.json.lz4 | unavailable | unavailable | 602.64 MB | 2019-10-26 |
trial1_ssh_stanford64_zmap.json.lz4 | unavailable | unavailable | 18.06 GB | 2020-09-15 |
trial1_tls_australia_zgrab.json.lz4 | unavailable | unavailable | 233.47 GB | 2019-10-22 |
trial1_tls_australia_zmap.json.lz4 | unavailable | unavailable | 14.12 GB | 2020-09-15 |
trial1_tls_brazil_zgrab.json.lz4 | unavailable | unavailable | 233.47 GB | 2019-10-22 |
trial1_tls_brazil_zmap.json.lz4 | unavailable | unavailable | 14.6 GB | 2020-09-15 |
trial1_tls_censys_zgrab.json.lz4 | unavailable | unavailable | 225.95 GB | 2019-10-22 |
trial1_tls_censys_zmap.json.lz4 | unavailable | unavailable | 13.69 GB | 2020-09-15 |
trial1_tls_germany_zgrab.json.lz4 | unavailable | unavailable | 232.36 GB | 2019-10-22 |
trial1_tls_germany_zmap.json.lz4 | unavailable | unavailable | 13.78 GB | 2020-09-15 |
trial1_tls_japan_zgrab.json.lz4 | unavailable | unavailable | 234.14 GB | 2019-10-22 |
trial1_tls_japan_zmap.json.lz4 | unavailable | unavailable | 14.34 GB | 2020-09-15 |
trial1_tls_stanford1_zgrab.json.lz4 | unavailable | unavailable | 231.52 GB | 2019-10-22 |
trial1_tls_stanford1_zmap.json.lz4 | unavailable | unavailable | 14.03 GB | 2020-09-15 |
trial1_tls_stanford64_zgrab.json.lz4 | unavailable | unavailable | 232.14 GB | 2019-10-22 |
trial1_tls_stanford64_zmap.json.lz4 | unavailable | unavailable | 19.38 GB | 2020-09-15 |
trial2_dns_brazil_zmap.json.lz4 | unavailable | unavailable | 24.02 GB | 2020-09-16 |
trial2_dns_censys_zmap.json.lz4 | unavailable | unavailable | 22.77 GB | 2020-09-16 |
trial2_dns_germany_zmap.json.lz4 | unavailable | unavailable | 23.95 GB | 2020-09-16 |
trial2_dns_japan_zmap.json.lz4 | unavailable | unavailable | 23.22 GB | 2020-09-16 |
trial2_dns_stanford1_zmap.json.lz4 | unavailable | unavailable | 23.42 GB | 2020-09-16 |
trial2_dns_stanford64_zmap.json.lz4 | unavailable | unavailable | 30.45 GB | 2020-09-16 |
trial2_http_australia_zgrab.json.lz4 | unavailable | unavailable | 159.91 GB | 2019-11-20 |
trial2_http_australia_zmap.json.lz4 | unavailable | unavailable | 13.68 GB | 2020-09-15 |
trial2_http_brazil_zgrab.json.lz4 | unavailable | unavailable | 159.59 GB | 2019-11-20 |
trial2_http_brazil_zmap.json.lz4 | unavailable | unavailable | 14.09 GB | 2020-09-15 |
trial2_http_censys_zgrab.json.lz4 | unavailable | unavailable | 153.61 GB | 2019-11-20 |
trial2_http_censys_zmap.json.lz4 | unavailable | unavailable | 13.09 GB | 2020-09-15 |
trial2_http_germany_zgrab.json.lz4 | unavailable | unavailable | 163.23 GB | 2019-11-20 |
trial2_http_germany_zmap.json.lz4 | unavailable | unavailable | 13.63 GB | 2020-09-15 |
trial2_http_japan_zgrab.json.lz4 | unavailable | unavailable | 160.64 GB | 2019-11-20 |
trial2_http_japan_zmap.json.lz4 | unavailable | unavailable | 13.88 GB | 2020-09-15 |
trial2_http_stanford1_zgrab.json.lz4 | unavailable | unavailable | 160.63 GB | 2019-11-20 |
trial2_http_stanford1_zmap.json.lz4 | unavailable | unavailable | 13.68 GB | 2020-09-15 |
trial2_http_stanford64_zgrab.json.lz4 | unavailable | unavailable | 160.9 GB | 2019-11-20 |
trial2_http_stanford64_zmap.json.lz4 | unavailable | unavailable | 18.72 GB | 2020-09-15 |
trial2_ssh_australia_zgrab.json.lz4 | unavailable | unavailable | 592.67 MB | 2019-11-28 |
trial2_ssh_australia_zmap.json.lz4 | unavailable | unavailable | 10.96 GB | 2020-09-16 |
trial2_ssh_brazil_zgrab.json.lz4 | unavailable | unavailable | 606.07 MB | 2019-11-28 |
trial2_ssh_brazil_zmap.json.lz4 | unavailable | unavailable | 12.87 GB | 2020-09-16 |
trial2_ssh_censys_zgrab.json.lz4 | unavailable | unavailable | 578.06 MB | 2019-11-28 |
trial2_ssh_censys_zmap.json.lz4 | unavailable | unavailable | 11.37 GB | 2020-09-16 |
trial2_ssh_germany_zgrab.json.lz4 | unavailable | unavailable | 656.72 MB | 2019-11-28 |
trial2_ssh_germany_zmap.json.lz4 | unavailable | unavailable | 11.69 GB | 2020-09-16 |
trial2_ssh_japan_zgrab.json.lz4 | unavailable | unavailable | 611.99 MB | 2019-11-28 |
trial2_ssh_japan_zmap.json.lz4 | unavailable | unavailable | 12.37 GB | 2020-09-16 |
trial2_ssh_stanford1_zgrab.json.lz4 | unavailable | unavailable | 597.92 MB | 2019-11-28 |
trial2_ssh_stanford1_zmap.json.lz4 | unavailable | unavailable | 12.45 GB | 2020-09-16 |
trial2_ssh_stanford64_zgrab.json.lz4 | unavailable | unavailable | 607.38 MB | 2019-11-28 |
trial2_ssh_stanford64_zmap.json.lz4 | unavailable | unavailable | 17.99 GB | 2020-09-16 |
trial2_tls_australia_zgrab.json.lz4 | unavailable | unavailable | 230.78 GB | 2019-11-26 |
trial2_tls_australia_zmap.json.lz4 | unavailable | unavailable | 13.55 GB | 2020-09-16 |
trial2_tls_brazil_zgrab.json.lz4 | unavailable | unavailable | 235.46 GB | 2019-11-26 |
trial2_tls_brazil_zmap.json.lz4 | unavailable | unavailable | 14.71 GB | 2020-09-16 |
trial2_tls_censys_zgrab.json.lz4 | unavailable | unavailable | 228.61 GB | 2019-11-26 |
trial2_tls_censys_zmap.json.lz4 | unavailable | unavailable | 13.64 GB | 2020-09-16 |
trial2_tls_germany_zgrab.json.lz4 | unavailable | unavailable | 238.96 GB | 2019-11-26 |
trial2_tls_germany_zmap.json.lz4 | unavailable | unavailable | 14.06 GB | 2020-09-16 |
trial2_tls_japan_zgrab.json.lz4 | unavailable | unavailable | 236.41 GB | 2019-11-26 |
trial2_tls_japan_zmap.json.lz4 | unavailable | unavailable | 14.22 GB | 2020-09-16 |
trial2_tls_stanford1_zgrab.json.lz4 | unavailable | unavailable | 235.98 GB | 2019-11-26 |
trial2_tls_stanford1_zmap.json.lz4 | unavailable | unavailable | 14.17 GB | 2020-09-16 |
trial2_tls_stanford64_zgrab.json.lz4 | unavailable | unavailable | 236.94 GB | 2019-11-26 |
trial2_tls_stanford64_zmap.json.lz4 | unavailable | unavailable | 19.58 GB | 2020-09-16 |
trial3_dns_australia_zmap.json.lz4 | unavailable | unavailable | 1.7 GB | 2020-09-16 |
trial3_dns_brazil_zmap.json.lz4 | unavailable | unavailable | 24.03 GB | 2020-09-16 |
trial3_dns_censys_zmap.json.lz4 | unavailable | unavailable | 23.44 GB | 2020-09-16 |
trial3_dns_germany_zmap.json.lz4 | unavailable | unavailable | 24.0 GB | 2020-09-16 |
trial3_dns_japan_zmap.json.lz4 | unavailable | unavailable | 23.23 GB | 2020-09-16 |
trial3_dns_stanford1_zmap.json.lz4 | unavailable | unavailable | 23.65 GB | 2020-09-16 |
trial3_dns_stanford64_zmap.json.lz4 | unavailable | unavailable | 30.85 GB | 2020-09-16 |
trial3_http_australia_zgrab.json.lz4 | unavailable | unavailable | 158.03 GB | 2019-12-12 |
trial3_http_australia_zmap.json.lz4 | unavailable | unavailable | 13.52 GB | 2020-09-16 |
trial3_http_brazil_zgrab.json.lz4 | unavailable | unavailable | 159.93 GB | 2019-12-12 |
trial3_http_brazil_zmap.json.lz4 | unavailable | unavailable | 14.46 GB | 2020-09-16 |
trial3_http_censys_zgrab.json.lz4 | unavailable | unavailable | 156.4 GB | 2019-12-12 |
trial3_http_censys_zmap.json.lz4 | unavailable | unavailable | 13.5 GB | 2020-09-16 |
trial3_http_germany_zgrab.json.lz4 | unavailable | unavailable | 163.94 GB | 2019-12-12 |
trial3_http_germany_zmap.json.lz4 | unavailable | unavailable | 13.85 GB | 2020-09-16 |
trial3_http_japan_zgrab.json.lz4 | unavailable | unavailable | 160.74 GB | 2019-12-12 |
trial3_http_japan_zmap.json.lz4 | unavailable | unavailable | 13.92 GB | 2020-09-16 |
trial3_http_stanford1_zgrab.json.lz4 | unavailable | unavailable | 161.61 GB | 2019-12-12 |
trial3_http_stanford1_zmap.json.lz4 | unavailable | unavailable | 13.97 GB | 2020-09-16 |
trial3_http_stanford64_zgrab.json.lz4 | unavailable | unavailable | 162.75 GB | 2019-12-12 |
trial3_http_stanford64_zmap.json.lz4 | unavailable | unavailable | 19.15 GB | 2020-09-16 |
trial3_ssh_australia_zgrab.json.lz4 | unavailable | unavailable | 612.52 MB | 2019-12-10 |
trial3_ssh_australia_zmap.json.lz4 | unavailable | unavailable | 11.11 GB | 2020-09-16 |
trial3_ssh_brazil_zgrab.json.lz4 | unavailable | unavailable | 618.59 MB | 2019-12-10 |
trial3_ssh_brazil_zmap.json.lz4 | unavailable | unavailable | 12.88 GB | 2020-09-16 |
trial3_ssh_censys_zgrab.json.lz4 | unavailable | unavailable | 591.77 MB | 2019-12-10 |
trial3_ssh_censys_zmap.json.lz4 | unavailable | unavailable | 11.6 GB | 2020-09-16 |
trial3_ssh_germany_zgrab.json.lz4 | unavailable | unavailable | 671.08 MB | 2019-12-10 |
trial3_ssh_germany_zmap.json.lz4 | unavailable | unavailable | 11.7 GB | 2020-09-16 |
trial3_ssh_japan_zgrab.json.lz4 | unavailable | unavailable | 619.04 MB | 2019-12-10 |
trial3_ssh_japan_zmap.json.lz4 | unavailable | unavailable | 12.11 GB | 2020-09-16 |
trial3_ssh_stanford1_zgrab.json.lz4 | unavailable | unavailable | 611.89 MB | 2019-12-10 |
trial3_ssh_stanford1_zmap.json.lz4 | unavailable | unavailable | 12.47 GB | 2020-09-16 |
trial3_ssh_stanford64_zgrab.json.lz4 | unavailable | unavailable | 623.59 MB | 2019-12-10 |
trial3_ssh_stanford64_zmap.json.lz4 | unavailable | unavailable | 18.02 GB | 2020-09-16 |
trial3_tls_australia_zgrab.json.lz4 | unavailable | unavailable | 233.45 GB | 2019-12-14 |
trial3_tls_australia_zmap.json.lz4 | unavailable | unavailable | 13.82 GB | 2020-09-16 |
trial3_tls_brazil_zgrab.json.lz4 | unavailable | unavailable | 235.25 GB | 2019-12-14 |
trial3_tls_brazil_zmap.json.lz4 | unavailable | unavailable | 14.51 GB | 2020-09-16 |
trial3_tls_censys_zgrab.json.lz4 | unavailable | unavailable | 231.85 GB | 2019-12-14 |
trial3_tls_censys_zmap.json.lz4 | unavailable | unavailable | 13.96 GB | 2020-09-16 |
trial3_tls_germany_zgrab.json.lz4 | unavailable | unavailable | 238.47 GB | 2019-12-14 |
trial3_tls_germany_zmap.json.lz4 | unavailable | unavailable | 14.09 GB | 2020-09-16 |
trial3_tls_japan_zgrab.json.lz4 | unavailable | unavailable | 235.64 GB | 2019-12-14 |
trial3_tls_japan_zmap.json.lz4 | unavailable | unavailable | 14.12 GB | 2020-09-16 |
trial3_tls_stanford1_zgrab.json.lz4 | unavailable | unavailable | 236.16 GB | 2019-12-14 |
trial3_tls_stanford1_zmap.json.lz4 | unavailable | unavailable | 14.17 GB | 2020-09-16 |
trial3_tls_stanford64_zgrab.json.lz4 | unavailable | unavailable | 237.26 GB | 2019-12-14 |
trial3_tls_stanford64_zmap.json.lz4 | unavailable | unavailable | 19.6 GB | 2020-09-16 |
trial4_http_australia_zgrab.json.lz4 | unavailable | unavailable | 159.88 GB | 2020-09-04 |
trial4_http_australia_zmap.json.lz4 | unavailable | unavailable | 12.91 GB | 2020-09-16 |
trial4_http_censys_zgrab.json.lz4 | unavailable | unavailable | 162.51 GB | 2020-09-05 |
trial4_http_censys_zmap.json.lz4 | unavailable | unavailable | 12.99 GB | 2020-09-16 |
trial4_http_germany_zgrab.json.lz4 | unavailable | unavailable | 163.63 GB | 2020-09-04 |
trial4_http_germany_zmap.json.lz4 | unavailable | unavailable | 12.83 GB | 2020-09-16 |
trial4_http_he_zgrab.json.lz4 | unavailable | unavailable | 164.76 GB | 2020-09-06 |
trial4_http_he_zmap.json.lz4 | unavailable | unavailable | 13.05 GB | 2020-09-16 |
trial4_http_japan_zgrab.json.lz4 | unavailable | unavailable | 163.43 GB | 2020-09-04 |
trial4_http_japan_zmap.json.lz4 | unavailable | unavailable | 13.05 GB | 2020-09-16 |
trial4_http_ntt_zgrab.json.lz4 | unavailable | unavailable | 164.42 GB | 2020-09-06 |
trial4_http_ntt_zmap.json.lz4 | unavailable | unavailable | 13.08 GB | 2020-09-16 |
trial4_http_stanford1_zgrab.json.lz4 | unavailable | unavailable | 162.49 GB | 2020-09-04 |
trial4_http_stanford1_zmap.json.lz4 | unavailable | unavailable | 13.02 GB | 2020-09-16 |
trial4_http_telia_zgrab.json.lz4 | unavailable | unavailable | 163.54 GB | 2020-09-06 |
trial4_http_telia_zmap.json.lz4 | unavailable | unavailable | 12.98 GB | 2020-09-16 |
trial5_http_australia_zgrab.json.lz4 | unavailable | unavailable | 159.1 GB | 2020-09-11 |
trial5_http_australia_zmap.json.lz4 | unavailable | unavailable | 12.89 GB | 2020-09-16 |
trial5_http_censys_zgrab.json.lz4 | unavailable | unavailable | 162.03 GB | 2020-09-11 |
trial5_http_censys_zmap.json.lz4 | unavailable | unavailable | 12.99 GB | 2020-09-16 |
trial5_http_germany_zgrab.json.lz4 | unavailable | unavailable | 163.33 GB | 2020-09-11 |
trial5_http_germany_zmap.json.lz4 | unavailable | unavailable | 12.81 GB | 2020-09-16 |
trial5_http_he_zgrab.json.lz4 | unavailable | unavailable | 163.61 GB | 2020-09-11 |
trial5_http_he_zmap.json.lz4 | unavailable | unavailable | 12.99 GB | 2020-09-16 |
trial5_http_japan_zgrab.json.lz4 | unavailable | unavailable | 162.94 GB | 2020-09-11 |
trial5_http_japan_zmap.json.lz4 | unavailable | unavailable | 13.05 GB | 2020-09-16 |
trial5_http_ntt_zgrab.json.lz4 | unavailable | unavailable | 162.5 GB | 2020-09-11 |
trial5_http_ntt_zmap.json.lz4 | unavailable | unavailable | 13.02 GB | 2020-09-16 |
trial5_http_stanford1_zgrab.json.lz4 | unavailable | unavailable | 162.08 GB | 2020-09-11 |
trial5_http_stanford1_zmap.json.lz4 | unavailable | unavailable | 13.02 GB | 2020-09-16 |
trial5_http_telia_zgrab.json.lz4 | unavailable | unavailable | 163.17 GB | 2020-09-11 |
trial5_http_telia_zmap.json.lz4 | unavailable | unavailable | 12.92 GB | 2020-09-16 |