AdGuard DNS v2.0 goes open source
AdGuard DNS has always been an open-source project in accordance with our policy (all our free products are open source).
But with the release of AdGuard DNS 2.0, we encountered a problem: on the one hand, this version provides a free public server, but on the other hand, it also has personal server functionality. We figured out what to do for a while, and after all we decided to open the code, but under a fairly strict license (AGPL).
Why are we doing that? There are several reasons for this decision.
First, it's essential for us that AdGuard DNS users trust us and see exactly what the server is doing. One can't just say that we are for all good and against all bad, the community should be able to check, verify and point out mistakes.
Second, we feel like we've done a good job and we'd love to share the results with the open source community. We hope that what we're doing can be useful to others and we can therefore contribute to further development of DNS.
At this point, we could just provide a link to our repo, but that would be boring and uninteresting. So today I'm going to tell you the story of how the AdGuard DNS evolved with time, what drove the changes, and why we made certain architectural decisions.
So what is AdGuard DNS today?
- It's deployed on dozens of servers worldwide
- It processes more than 1,000,000 DNS requests per second from over 50 million users
- 90% of these requests are encrypted protocols (DNS-over-TLS, DNS-over-HTTPS, DNS-over-QUIC).
By the numbers, you can tell that this is a very serious workload, and I think we've coped with it pretty well. But when we just started it was not so rosy.
Warning: This is a very technical post aimed at engineers and their supporters. Be prepared.
How it started
It was back in 2016. AdGuard Ad Blocker had already been launched on every platform where it could be run, but it still had some blind spots. And with the growth of IOT, those blind spots were proliferating rapidly. How could we get to them in a way that was available to a casual user?
At that time all we knew about DNS was that it had to answer the question about what the IP address of this or that domain was. But still, it occurred to us that DNS is the simplest solution to achieve content blocking everywhere and not just on the devices you can install AdGuard on. Setting it up is not that complicated, you just enter four numbers into the router settings and voila, the whole network is protected. So once we decided on what we’re doing, it was just a matter of launching this service and making it public.
The task seemed elementary:
- Our server receives a DNS query
- Depending on which domain is requested, we do one of two things:
- Either it passes the query further to an instance of a "recursive DNS server" (in the first weeks we were simply sending it to Google DNS, setting up our own instance only later).
- Or immediately respond that such a domain does not exist. In this case the client will not be able to connect to the domain and the blocking task will be completed.
We didn't have much time to spend on a pet project so we used the tools we knew and tried to reuse what's already there. At that time AdGuard for Android was mostly written in Java so it was natural to use Java for writing the DNS service and reuse some of its filtering code.
The first beta version in Java
Listening incoming UDP packets on port 53 and send them to be processed in a separate stream:
Parsing incoming requests and then either passing the same UDP packet on, or returning a blocked response:
I remember how happy we were that the whole process took a couple of weeks to accomplish. But while preparing for this post I dug out the old code and, to be honest, the number of purely functional mistakes made during writing the code was off the charts.
For example, here is one of them: the server listened only to UDP. With DNS it is strictly forbidden to do so. The thing is that UDP limits the size of a DNS response. If this size exceeds a certain value (e.g. more than 1252 bytes), the server will return a truncated answer to the client, which is a signal to use TCP for the full response. Only, how would the client use it?
DNS filtering rules
In the very beginning of AdGuard DNS we made one good decision I am still happy about. Historically, DNS content blocking relied on using huge HOSTS files, simple lists of domain names to be blocked. But there are so many things that can be done on the DNS-level besides just “blocking”, and you need a way to express these more complicated rules. We decided to adopt familiar “adblock-style” syntax for DNS blocklists and extend it with additional DNS-specific capabilities.
Let me show you a couple of examples of what you can do with it (note that it works in all AdGuard products including AdGuard Home).
||example.org^$client=MyMac: blocks every query from the device “MyMac”||example.org^$dnsrewrite=example.net: redirects example.org to example.net*$denyallow=com|net: blocks everything save for *.com and *.net domains*$dnstype=AAAA: gets rid of IPv6
If you’re interested, there’s more information about it in the documentation.
AdGuard DNS v1.0
But even in spite of the stupid mistakes we made in the first beta version, AdGuard DNS became popular and we started to devote more and more time to it. Until one day we decided to rewrite it completely.
That's how AdGuard DNS v1.0 was born. We took CoreDNS, an extensible DNS server written in Go, as the basis. The reason we chose it was its flexible plugin system, which made it easy to extend its functionality and add filtering capabilities, which is what AdGuard DNS was designed for. To do that, we rewrote the filtering engine in Go (and naturally, it's also open-source and published on Github).
But soon enough we faced the problem of expanding CoreDNS further. Despite the flexibility of the plugin mechanism, not everything could be implemented with it. To speed up the work of DNS-over-TLS and to add native support for DNSCrypt and DNS-over-QUIC, we decided to support our own fork of CoreDNS, but the further it went, the harder it was to synchronize it with the upstream. So when we were developing the new version of AdGuard DNS, we decided to write the implementation of the DNS server from scratch.
AdGuard DNS v2.0
AdGuard DNS v2.0 greatly enhances the user experience compared to a regular DNS server. It allows users to manage filtering themselves and monitor what their devices are doing.
Personal dashboard in AdGuard DNS v2.0
And this has made the service architecture much more complex. An approximate scheme of the current configuration is shown on the diagram below (it includes only the essential parts):
Working parts of AdGuard DNS v2.0
- DNS Servers (1,2… n) are instances of the DNS server itself, responsible for processing and filtering requests. What they also do:
- Periodically request updates of client configuration from API Server
- Record log requests on a disk for registered clients with logging enabled
- Using Filebeat and Logstash, the query log is quickly transferred to the central analytical database.
- We use Clickhouse to store logs and build reports on them. In our opinion, this is the best analytics database available.
- Web Server is the AdGuard DNS website. Web Server does not communicate with databases directly, but does so via the API Server.
- API Server provides other services and users with an API that allows them to control settings and access information from databases.
Today we are publishing the code of the main part of AdGuard DNS — the DNS server itself. You can find it on Github.
In the future, we also plan to use some of this code in our other open-source projects (AdGuard Home and dnsproxy) and use a more permissive license for these parts.
