PDR Labs Internet Mapping Experiment

PDR Labs is engaged in an ongoing experiment to continuously map the Internet in its entirety. Our crawls are not malicious or invasive, and are intended to understand what addresses on the Internet-at-large are in use. If you have additional questions, please contact research@pdrlabs.net.

What are you doing?

We are an industry research group focused on understanding large-scale traffic patterns, and so it is useful to understand what address space is being actively utilized. To accomplish this, our crawling network makes HTTP HEAD requests on the public ports 80 and 443 (web traffic) for IPv4 and sections of IPv6 address space. This lets us see which addresses are actively being used in large traffic patterns. We respect all robots.txt entries.

Why are you doing this?

While there are academic projects (such as Censys.io) that do effectively the same thing that we’re doing, these projects have limits of use on the data. We are an industry group using this data to help build products and services that are used to manage and monitor the performance and security of large-scale networks. Because we build commercial tools, we’re unable to take advantage of existing datasets and have undertaken this project which uses the same techniques and has similar goals to academic projects.

I’d prefer my hosts not be included, what can I do?

While we respect and honor any robots.txt directives, if you’d prefer to contact us directly and ask that your addresses be blacklisted from our crawling, please contact blacklist@pdrlabs.net.