Fast IPv4 scanning has enabled researchers to answer a wealth of security and networking questions. Yet, despite widespread use, there has been little validation of the methodology’s accuracy, including whether a single scan provides sufficient coverage. In this paper, we analyze how scan origin affects the results of Internet-wide scans by completing three HTTP, HTTPS, and SSH scans from seven geographically and topologically diverse networks. We find that individual origins miss an average 1.6–8.4% of HTTP, 1.5–4.6% of HTTPS, and 8.3–18.2% of SSH hosts. We analyze why origins see different hosts, and show how permanent and temporary blocking, packet loss, geographic biases, and transient outages affect scan results. We discuss the implications for scanning and provide recommendations for future studies.


Study Details

Study
On the Origin of Scanning: The Impact of Location on Internet-Wide Scans
ACM Internet Measurement Conference (IMC), October 2020
Authors
Gerry Wan, Liz Izhikevich, David Adrian, Katsunari Yoshioka, Ralph Holz, Christian Rossow, Zakir Durumeric
Contact
Gerry Wan

Dataset Details

We provide access to the raw ZMap and ZGrab scans that are run from each network perspectives across three trials.


File Download

File NameMetaDataSHA-1 FingerprintSizeUpdated At
trial1_dns_australia_zmap.json.lz4 unavailable unavailable 22.8 GB 2020-09-15
trial1_dns_brazil_zmap.json.lz4 unavailable unavailable 23.33 GB 2020-09-15
trial1_dns_censys_zmap.json.lz4 unavailable unavailable 22.6 GB 2020-09-15
trial1_dns_germany_zmap.json.lz4 unavailable unavailable 23.24 GB 2020-09-15
trial1_dns_japan_zmap.json.lz4 unavailable unavailable 22.72 GB 2020-09-15
trial1_dns_stanford1_zmap.json.lz4 unavailable unavailable 22.99 GB 2020-09-15
trial1_dns_stanford64_zmap.json.lz4 unavailable unavailable 30.09 GB 2020-09-15
trial1_http_australia_zgrab.json.lz4 unavailable unavailable 155.95 GB 2019-10-25
trial1_http_australia_zmap.json.lz4 unavailable unavailable 13.74 GB 2020-09-15
trial1_http_brazil_zgrab.json.lz4 unavailable unavailable 157.01 GB 2019-10-25
trial1_http_brazil_zmap.json.lz4 unavailable unavailable 14.21 GB 2020-09-15
trial1_http_censys_zgrab.json.lz4 unavailable unavailable 150.7 GB 2019-10-24
trial1_http_censys_zmap.json.lz4 unavailable unavailable 13.22 GB 2020-09-15
trial1_http_germany_zgrab.json.lz4 unavailable unavailable 161.42 GB 2019-10-24
trial1_http_germany_zmap.json.lz4 unavailable unavailable 13.6 GB 2020-09-15
trial1_http_japan_zgrab.json.lz4 unavailable unavailable 158.56 GB 2019-10-24
trial1_http_japan_zmap.json.lz4 unavailable unavailable 14.0 GB 2020-09-15
trial1_http_stanford1_zgrab.json.lz4 unavailable unavailable 158.25 GB 2019-10-24
trial1_http_stanford1_zmap.json.lz4 unavailable unavailable 13.79 GB 2020-09-15
trial1_http_stanford64_zgrab.json.lz4 unavailable unavailable 158.49 GB 2019-10-24
trial1_http_stanford64_zmap.json.lz4 unavailable unavailable 18.88 GB 2020-09-15
trial1_ssh_australia_zgrab.json.lz4 unavailable unavailable 594.47 MB 2019-10-26
trial1_ssh_australia_zmap.json.lz4 unavailable unavailable 11.29 GB 2020-09-15
trial1_ssh_brazil_zgrab.json.lz4 unavailable unavailable 602.51 MB 2019-10-26
trial1_ssh_brazil_zmap.json.lz4 unavailable unavailable 12.79 GB 2020-09-15
trial1_ssh_censys_zgrab.json.lz4 unavailable unavailable 575.88 MB 2019-10-26
trial1_ssh_censys_zmap.json.lz4 unavailable unavailable 11.46 GB 2020-09-15
trial1_ssh_germany_zgrab.json.lz4 unavailable unavailable 647.61 MB 2019-10-26
trial1_ssh_germany_zmap.json.lz4 unavailable unavailable 11.61 GB 2020-09-15
trial1_ssh_japan_zgrab.json.lz4 unavailable unavailable 607.1 MB 2019-10-26
trial1_ssh_japan_zmap.json.lz4 unavailable unavailable 12.49 GB 2020-09-15
trial1_ssh_stanford1_zgrab.json.lz4 unavailable unavailable 586.43 MB 2019-10-26
trial1_ssh_stanford1_zmap.json.lz4 unavailable unavailable 12.48 GB 2020-09-15
trial1_ssh_stanford64_zgrab.json.lz4 unavailable unavailable 602.64 MB 2019-10-26
trial1_ssh_stanford64_zmap.json.lz4 unavailable unavailable 18.06 GB 2020-09-15
trial1_tls_australia_zgrab.json.lz4 unavailable unavailable 233.47 GB 2019-10-22
trial1_tls_australia_zmap.json.lz4 unavailable unavailable 14.12 GB 2020-09-15
trial1_tls_brazil_zgrab.json.lz4 unavailable unavailable 233.47 GB 2019-10-22
trial1_tls_brazil_zmap.json.lz4 unavailable unavailable 14.6 GB 2020-09-15
trial1_tls_censys_zgrab.json.lz4 unavailable unavailable 225.95 GB 2019-10-22
trial1_tls_censys_zmap.json.lz4 unavailable unavailable 13.69 GB 2020-09-15
trial1_tls_germany_zgrab.json.lz4 unavailable unavailable 232.36 GB 2019-10-22
trial1_tls_germany_zmap.json.lz4 unavailable unavailable 13.78 GB 2020-09-15
trial1_tls_japan_zgrab.json.lz4 unavailable unavailable 234.14 GB 2019-10-22
trial1_tls_japan_zmap.json.lz4 unavailable unavailable 14.34 GB 2020-09-15
trial1_tls_stanford1_zgrab.json.lz4 unavailable unavailable 231.52 GB 2019-10-22
trial1_tls_stanford1_zmap.json.lz4 unavailable unavailable 14.03 GB 2020-09-15
trial1_tls_stanford64_zgrab.json.lz4 unavailable unavailable 232.14 GB 2019-10-22
trial1_tls_stanford64_zmap.json.lz4 unavailable unavailable 19.38 GB 2020-09-15
trial2_dns_brazil_zmap.json.lz4 unavailable unavailable 24.02 GB 2020-09-16
trial2_dns_censys_zmap.json.lz4 unavailable unavailable 22.77 GB 2020-09-16
trial2_dns_germany_zmap.json.lz4 unavailable unavailable 23.95 GB 2020-09-16
trial2_dns_japan_zmap.json.lz4 unavailable unavailable 23.22 GB 2020-09-16
trial2_dns_stanford1_zmap.json.lz4 unavailable unavailable 23.42 GB 2020-09-16
trial2_dns_stanford64_zmap.json.lz4 unavailable unavailable 30.45 GB 2020-09-16
trial2_http_australia_zgrab.json.lz4 unavailable unavailable 159.91 GB 2019-11-20
trial2_http_australia_zmap.json.lz4 unavailable unavailable 13.68 GB 2020-09-15
trial2_http_brazil_zgrab.json.lz4 unavailable unavailable 159.59 GB 2019-11-20
trial2_http_brazil_zmap.json.lz4 unavailable unavailable 14.09 GB 2020-09-15
trial2_http_censys_zgrab.json.lz4 unavailable unavailable 153.61 GB 2019-11-20
trial2_http_censys_zmap.json.lz4 unavailable unavailable 13.09 GB 2020-09-15
trial2_http_germany_zgrab.json.lz4 unavailable unavailable 163.23 GB 2019-11-20
trial2_http_germany_zmap.json.lz4 unavailable unavailable 13.63 GB 2020-09-15
trial2_http_japan_zgrab.json.lz4 unavailable unavailable 160.64 GB 2019-11-20
trial2_http_japan_zmap.json.lz4 unavailable unavailable 13.88 GB 2020-09-15
trial2_http_stanford1_zgrab.json.lz4 unavailable unavailable 160.63 GB 2019-11-20
trial2_http_stanford1_zmap.json.lz4 unavailable unavailable 13.68 GB 2020-09-15
trial2_http_stanford64_zgrab.json.lz4 unavailable unavailable 160.9 GB 2019-11-20
trial2_http_stanford64_zmap.json.lz4 unavailable unavailable 18.72 GB 2020-09-15
trial2_ssh_australia_zgrab.json.lz4 unavailable unavailable 592.67 MB 2019-11-28
trial2_ssh_australia_zmap.json.lz4 unavailable unavailable 10.96 GB 2020-09-16
trial2_ssh_brazil_zgrab.json.lz4 unavailable unavailable 606.07 MB 2019-11-28
trial2_ssh_brazil_zmap.json.lz4 unavailable unavailable 12.87 GB 2020-09-16
trial2_ssh_censys_zgrab.json.lz4 unavailable unavailable 578.06 MB 2019-11-28
trial2_ssh_censys_zmap.json.lz4 unavailable unavailable 11.37 GB 2020-09-16
trial2_ssh_germany_zgrab.json.lz4 unavailable unavailable 656.72 MB 2019-11-28
trial2_ssh_germany_zmap.json.lz4 unavailable unavailable 11.69 GB 2020-09-16
trial2_ssh_japan_zgrab.json.lz4 unavailable unavailable 611.99 MB 2019-11-28
trial2_ssh_japan_zmap.json.lz4 unavailable unavailable 12.37 GB 2020-09-16
trial2_ssh_stanford1_zgrab.json.lz4 unavailable unavailable 597.92 MB 2019-11-28
trial2_ssh_stanford1_zmap.json.lz4 unavailable unavailable 12.45 GB 2020-09-16
trial2_ssh_stanford64_zgrab.json.lz4 unavailable unavailable 607.38 MB 2019-11-28
trial2_ssh_stanford64_zmap.json.lz4 unavailable unavailable 17.99 GB 2020-09-16
trial2_tls_australia_zgrab.json.lz4 unavailable unavailable 230.78 GB 2019-11-26
trial2_tls_australia_zmap.json.lz4 unavailable unavailable 13.55 GB 2020-09-16
trial2_tls_brazil_zgrab.json.lz4 unavailable unavailable 235.46 GB 2019-11-26
trial2_tls_brazil_zmap.json.lz4 unavailable unavailable 14.71 GB 2020-09-16
trial2_tls_censys_zgrab.json.lz4 unavailable unavailable 228.61 GB 2019-11-26
trial2_tls_censys_zmap.json.lz4 unavailable unavailable 13.64 GB 2020-09-16
trial2_tls_germany_zgrab.json.lz4 unavailable unavailable 238.96 GB 2019-11-26
trial2_tls_germany_zmap.json.lz4 unavailable unavailable 14.06 GB 2020-09-16
trial2_tls_japan_zgrab.json.lz4 unavailable unavailable 236.41 GB 2019-11-26
trial2_tls_japan_zmap.json.lz4 unavailable unavailable 14.22 GB 2020-09-16
trial2_tls_stanford1_zgrab.json.lz4 unavailable unavailable 235.98 GB 2019-11-26
trial2_tls_stanford1_zmap.json.lz4 unavailable unavailable 14.17 GB 2020-09-16
trial2_tls_stanford64_zgrab.json.lz4 unavailable unavailable 236.94 GB 2019-11-26
trial2_tls_stanford64_zmap.json.lz4 unavailable unavailable 19.58 GB 2020-09-16
trial3_dns_australia_zmap.json.lz4 unavailable unavailable 1.7 GB 2020-09-16
trial3_dns_brazil_zmap.json.lz4 unavailable unavailable 24.03 GB 2020-09-16
trial3_dns_censys_zmap.json.lz4 unavailable unavailable 23.44 GB 2020-09-16
trial3_dns_germany_zmap.json.lz4 unavailable unavailable 24.0 GB 2020-09-16
trial3_dns_japan_zmap.json.lz4 unavailable unavailable 23.23 GB 2020-09-16
trial3_dns_stanford1_zmap.json.lz4 unavailable unavailable 23.65 GB 2020-09-16
trial3_dns_stanford64_zmap.json.lz4 unavailable unavailable 30.85 GB 2020-09-16
trial3_http_australia_zgrab.json.lz4 unavailable unavailable 158.03 GB 2019-12-12
trial3_http_australia_zmap.json.lz4 unavailable unavailable 13.52 GB 2020-09-16
trial3_http_brazil_zgrab.json.lz4 unavailable unavailable 159.93 GB 2019-12-12
trial3_http_brazil_zmap.json.lz4 unavailable unavailable 14.46 GB 2020-09-16
trial3_http_censys_zgrab.json.lz4 unavailable unavailable 156.4 GB 2019-12-12
trial3_http_censys_zmap.json.lz4 unavailable unavailable 13.5 GB 2020-09-16
trial3_http_germany_zgrab.json.lz4 unavailable unavailable 163.94 GB 2019-12-12
trial3_http_germany_zmap.json.lz4 unavailable unavailable 13.85 GB 2020-09-16
trial3_http_japan_zgrab.json.lz4 unavailable unavailable 160.74 GB 2019-12-12
trial3_http_japan_zmap.json.lz4 unavailable unavailable 13.92 GB 2020-09-16
trial3_http_stanford1_zgrab.json.lz4 unavailable unavailable 161.61 GB 2019-12-12
trial3_http_stanford1_zmap.json.lz4 unavailable unavailable 13.97 GB 2020-09-16
trial3_http_stanford64_zgrab.json.lz4 unavailable unavailable 162.75 GB 2019-12-12
trial3_http_stanford64_zmap.json.lz4 unavailable unavailable 19.15 GB 2020-09-16
trial3_ssh_australia_zgrab.json.lz4 unavailable unavailable 612.52 MB 2019-12-10
trial3_ssh_australia_zmap.json.lz4 unavailable unavailable 11.11 GB 2020-09-16
trial3_ssh_brazil_zgrab.json.lz4 unavailable unavailable 618.59 MB 2019-12-10
trial3_ssh_brazil_zmap.json.lz4 unavailable unavailable 12.88 GB 2020-09-16
trial3_ssh_censys_zgrab.json.lz4 unavailable unavailable 591.77 MB 2019-12-10
trial3_ssh_censys_zmap.json.lz4 unavailable unavailable 11.6 GB 2020-09-16
trial3_ssh_germany_zgrab.json.lz4 unavailable unavailable 671.08 MB 2019-12-10
trial3_ssh_germany_zmap.json.lz4 unavailable unavailable 11.7 GB 2020-09-16
trial3_ssh_japan_zgrab.json.lz4 unavailable unavailable 619.04 MB 2019-12-10
trial3_ssh_japan_zmap.json.lz4 unavailable unavailable 12.11 GB 2020-09-16
trial3_ssh_stanford1_zgrab.json.lz4 unavailable unavailable 611.89 MB 2019-12-10
trial3_ssh_stanford1_zmap.json.lz4 unavailable unavailable 12.47 GB 2020-09-16
trial3_ssh_stanford64_zgrab.json.lz4 unavailable unavailable 623.59 MB 2019-12-10
trial3_ssh_stanford64_zmap.json.lz4 unavailable unavailable 18.02 GB 2020-09-16
trial3_tls_australia_zgrab.json.lz4 unavailable unavailable 233.45 GB 2019-12-14
trial3_tls_australia_zmap.json.lz4 unavailable unavailable 13.82 GB 2020-09-16
trial3_tls_brazil_zgrab.json.lz4 unavailable unavailable 235.25 GB 2019-12-14
trial3_tls_brazil_zmap.json.lz4 unavailable unavailable 14.51 GB 2020-09-16
trial3_tls_censys_zgrab.json.lz4 unavailable unavailable 231.85 GB 2019-12-14
trial3_tls_censys_zmap.json.lz4 unavailable unavailable 13.96 GB 2020-09-16
trial3_tls_germany_zgrab.json.lz4 unavailable unavailable 238.47 GB 2019-12-14
trial3_tls_germany_zmap.json.lz4 unavailable unavailable 14.09 GB 2020-09-16
trial3_tls_japan_zgrab.json.lz4 unavailable unavailable 235.64 GB 2019-12-14
trial3_tls_japan_zmap.json.lz4 unavailable unavailable 14.12 GB 2020-09-16
trial3_tls_stanford1_zgrab.json.lz4 unavailable unavailable 236.16 GB 2019-12-14
trial3_tls_stanford1_zmap.json.lz4 unavailable unavailable 14.17 GB 2020-09-16
trial3_tls_stanford64_zgrab.json.lz4 unavailable unavailable 237.26 GB 2019-12-14
trial3_tls_stanford64_zmap.json.lz4 unavailable unavailable 19.6 GB 2020-09-16
trial4_http_australia_zgrab.json.lz4 unavailable unavailable 159.88 GB 2020-09-04
trial4_http_australia_zmap.json.lz4 unavailable unavailable 12.91 GB 2020-09-16
trial4_http_censys_zgrab.json.lz4 unavailable unavailable 162.51 GB 2020-09-05
trial4_http_censys_zmap.json.lz4 unavailable unavailable 12.99 GB 2020-09-16
trial4_http_germany_zgrab.json.lz4 unavailable unavailable 163.63 GB 2020-09-04
trial4_http_germany_zmap.json.lz4 unavailable unavailable 12.83 GB 2020-09-16
trial4_http_he_zgrab.json.lz4 unavailable unavailable 164.76 GB 2020-09-06
trial4_http_he_zmap.json.lz4 unavailable unavailable 13.05 GB 2020-09-16
trial4_http_japan_zgrab.json.lz4 unavailable unavailable 163.43 GB 2020-09-04
trial4_http_japan_zmap.json.lz4 unavailable unavailable 13.05 GB 2020-09-16
trial4_http_ntt_zgrab.json.lz4 unavailable unavailable 164.42 GB 2020-09-06
trial4_http_ntt_zmap.json.lz4 unavailable unavailable 13.08 GB 2020-09-16
trial4_http_stanford1_zgrab.json.lz4 unavailable unavailable 162.49 GB 2020-09-04
trial4_http_stanford1_zmap.json.lz4 unavailable unavailable 13.02 GB 2020-09-16
trial4_http_telia_zgrab.json.lz4 unavailable unavailable 163.54 GB 2020-09-06
trial4_http_telia_zmap.json.lz4 unavailable unavailable 12.98 GB 2020-09-16
trial5_http_australia_zgrab.json.lz4 unavailable unavailable 159.1 GB 2020-09-11
trial5_http_australia_zmap.json.lz4 unavailable unavailable 12.89 GB 2020-09-16
trial5_http_censys_zgrab.json.lz4 unavailable unavailable 162.03 GB 2020-09-11
trial5_http_censys_zmap.json.lz4 unavailable unavailable 12.99 GB 2020-09-16
trial5_http_germany_zgrab.json.lz4 unavailable unavailable 163.33 GB 2020-09-11
trial5_http_germany_zmap.json.lz4 unavailable unavailable 12.81 GB 2020-09-16
trial5_http_he_zgrab.json.lz4 unavailable unavailable 163.61 GB 2020-09-11
trial5_http_he_zmap.json.lz4 unavailable unavailable 12.99 GB 2020-09-16
trial5_http_japan_zgrab.json.lz4 unavailable unavailable 162.94 GB 2020-09-11
trial5_http_japan_zmap.json.lz4 unavailable unavailable 13.05 GB 2020-09-16
trial5_http_ntt_zgrab.json.lz4 unavailable unavailable 162.5 GB 2020-09-11
trial5_http_ntt_zmap.json.lz4 unavailable unavailable 13.02 GB 2020-09-16
trial5_http_stanford1_zgrab.json.lz4 unavailable unavailable 162.08 GB 2020-09-11
trial5_http_stanford1_zmap.json.lz4 unavailable unavailable 13.02 GB 2020-09-16
trial5_http_telia_zgrab.json.lz4 unavailable unavailable 163.17 GB 2020-09-11
trial5_http_telia_zmap.json.lz4 unavailable unavailable 12.92 GB 2020-09-16