URLs, DNS and TLS

Before a single byte of HTTP is sent, three things must happen: the URL is parsed, the hostname is resolved to an IP address, and a connection is established and encrypted. Each step has its own protocol, its own caching layer, and its own set of failures.

URL anatomy

A URL is a URI that also tells you how to reach the thing. The full grammar:

 scheme  userinfo      host        port  path        query        fragment
   │        │            │           │     │            │             │
 ┌─┴─┐ ┌────┴────┐ ┌─────┴─────┐  ┌──┴─┐ ┌─┴──┐ ┌───────┴───────┐ ┌───┴───┐
 https://ada:pw123@api.example.com:8443/v2/users?role=admin&page=2#results
       └─┬─┘
      separator

Component	Notes
Scheme	https, http, ws, wss, mailto, file, data. Decides the protocol. Case-insensitive, always written lowercase.
Userinfo	user:password@. Deprecated for HTTP — browsers strip it, and it leaks credentials. Still used by some non-HTTP schemes.
Host	A domain name, an IPv4 literal (93.184.215.14), or an IPv6 literal in brackets ([2606:2800:21f:cb07::]).
Port	Optional. Defaults from the scheme: 80 for http, 443 for https.
Path	Hierarchical, /-separated. Case-sensitivity depends entirely on the server.
Query	Everything after ?. Conventionally key=value pairs joined by &, but that convention is not in the URL spec — the server decides.
Fragment	Everything after #. Never sent to the server. Purely client-side: it scrolls to an element or drives a client-side router.

Note

The fragment being client-only is load-bearing. OAuth implicit flow put tokens in the fragment specifically so they never reached a server log. It is also why # in a URL you paste into an API call silently truncates your data.

Origin vs site

Two terms that look interchangeable and are not:

• An origin is scheme + host + port. https://app.example.com and https://app.example.com:8443 are different origins. This is what the same-origin policy uses.
• A site is roughly scheme + registrable domain (eTLD+1). app.example.com and api.example.com are the same site, different origins. SameSite cookies use this.

Relative URLs

A relative reference is resolved against a base URL:

Reference	Resolved against https://example.com/a/b/page
other	https://example.com/a/b/other
/other	https://example.com/other
../other	https://example.com/a/other
//cdn.example.com/x	https://cdn.example.com/x (protocol-relative — inherits the scheme)
?q=1	https://example.com/a/b/page?q=1
#top	https://example.com/a/b/page#top

Percent-encoding

URLs may only contain a limited ASCII set. Everything else — spaces, non-ASCII characters, and any reserved character used as data rather than as a delimiter — is encoded as % followed by two hex digits of its UTF-8 bytes.

Reserved characters, which have structural meaning: : / ? # [ ] @ ! $ & ' ( ) * + , ; =

Character	Encoded
space	%20 (or + inside a query string only)
/	%2F
?	%3F
#	%23
&	%26
=	%3D
+	%2B
é	%C3%A9 (two UTF-8 bytes)

// Encodes everything unsafe EXCEPT the structural characters : / ? # [ ] @ & = + $ , ;
encodeURI('https://example.com/a b?q=c&d=é');
// => 'https://example.com/a%20b?q=c&d=%C3%A9'

// Encodes structural characters too — use this for individual values
encodeURIComponent('a&b=c/d');
// => 'a%26b%3Dc%2Fd'

The rule: encodeURIComponent for each value, never for a whole URL. In practice, let a URL library build it:

const url = new URL('https://api.example.com/search');
url.searchParams.set('q', 'rust & go');
url.searchParams.set('page', '2');
url.toString();
// => 'https://api.example.com/search?q=rust+%26+go&page=2'

# curl --get --data-urlencode does the same thing from the shell
curl -v --get --data-urlencode 'q=rust & go' https://httpbin.org/get

Caution

+ means a literal plus in a path but a space in a query string, because HTML form encoding (application/x-www-form-urlencoded) predates and diverges from the URL spec. Encode literal plus signs as %2B in query values.

Internationalised domains

Hostnames on the wire are ASCII. münchen.de is converted by Punycode to xn--mnchen-3ya.de before DNS lookup. Browsers display the Unicode form but will fall back to showing the raw Punycode when the script mix looks like a homograph attack.

DNS

DNS maps names to addresses. It is a globally distributed, hierarchical, cached key-value store, and it runs over UDP port 53 (falling back to TCP for large answers).

The resolution chain

Reading www.example.com right to left — the hierarchy runs from the root down:

1. Stub resolver — the OS library in your application. Checks the local cache and /etc/hosts.
2. Recursive resolver — your ISP’s, or 1.1.1.1 / 8.8.8.8, or your router. Does the actual work and caches results for everyone behind it.
3. Root servers (.) — 13 logical addresses, heavily anycast. They answer: “ask the .com servers.”
4. TLD servers (.com) — answer: “ask ns1.example.com for example.com.”
5. Authoritative servers — hold the real zone file and give the final answer.

The resolver then caches that answer for its TTL and returns it. Steps 3–5 usually never happen for popular domains, because something in the chain already has the answer cached.

# Watch the full delegation chain, one step at a time
dig +trace example.com

Record types

Type	Holds	Example
A	An IPv4 address	example.com. 300 IN A 93.184.215.14
AAAA	An IPv6 address	example.com. 300 IN AAAA 2606:2800:21f:cb07::
CNAME	An alias to another name	www.example.com. IN CNAME example.com.
MX	Mail servers, with a priority number	example.com. IN MX 10 mail.example.com.
TXT	Arbitrary text. Used for SPF, DKIM, and domain-ownership proofs.	example.com. IN TXT "v=spf1 include:_spf.google.com ~all"
NS	The authoritative nameservers for a zone	example.com. IN NS a.iana-servers.net.
SOA	Zone metadata: primary NS, admin email, serial, timers
PTR	Reverse lookup — IP back to a name
CAA	Which certificate authorities may issue certs for this domain
SRV	Service location: host + port for a protocol
HTTPS / SVCB	Connection hints (ALPN, IP hints) so a client can go straight to HTTP/3

Caution

A CNAME cannot coexist with any other record at the same name, which is why you cannot put a CNAME on a zone apex (example.com itself) — the apex must hold SOA and NS. Providers work around this with non-standard ALIAS/ANAME/flattened-CNAME records that resolve server-side.

TTL

Every record carries a time to live in seconds. Resolvers must not serve it beyond that. A 300-second TTL means a change propagates within 5 minutes; an 86400 TTL means a day.

The operational pattern before a migration: lower the TTL to 60 a day ahead, make the change, verify, then raise it back. Raising it early does nothing — caches still hold the old, long TTL.

Tools

# Short answer only
dig +short example.com A

# Full response with sections and TTLs
dig example.com

# A specific record type
dig example.com MX
dig example.com TXT

# Ask a specific resolver, bypassing your local one
dig @1.1.1.1 example.com

# Which nameservers are authoritative
dig example.com NS +short

# Reverse lookup
dig -x 93.184.215.14

# Older, simpler tool — still everywhere
nslookup example.com
nslookup -type=MX example.com

Reading dig output: the ANSWER SECTION holds the records, the number before IN is the remaining TTL, and the flags: line tells you whether the answer was authoritative (aa) or came from a cache.

Encrypted DNS

Classic DNS is plaintext UDP — any network observer sees every hostname you look up. DoH (DNS over HTTPS, port 443) and DoT (DNS over TLS, port 853) encrypt the query to the resolver. Browsers ship DoH and may use it instead of the OS resolver, which is a common reason /etc/hosts edits appear not to work in Chrome or Firefox.

The TCP handshake

With an IP address in hand, the client opens a TCP connection. Three packets:

Client                          Server
  │ ── SYN (seq=x) ─────────────► │   "I want to talk, my sequence starts at x"
  │ ◄── SYN-ACK (seq=y, ack=x+1) ─│   "OK, mine starts at y, I got yours"
  │ ── ACK (ack=y+1) ───────────► │   "Got yours. Connection open."

One full round trip before any application data moves. That round trip is pure latency — on a 100 ms link it costs 100 ms no matter how fast either machine is. This is why connection reuse (keep-alive, HTTP/2 multiplexing) is one of the highest-leverage performance wins available.

TCP then provides an ordered, reliable byte stream: sequence numbers detect loss, ACKs confirm delivery, retransmission fixes gaps, and congestion control (slow start, etc.) throttles the sender. HTTP just writes bytes and trusts all of that.

# Is a port open and reachable?
nc -vz example.com 443

# Watch the handshake itself (needs root)
sudo tcpdump -n 'host example.com and tcp port 443' -c 6

TLS

TLS (Transport Layer Security — the successor to SSL, though “SSL” persists in tool names) wraps the TCP stream. HTTP over TLS is HTTPS.

What TLS provides

Three guarantees, and it is worth keeping them separate:

• Confidentiality — the traffic is encrypted; an observer sees ciphertext.
• Integrity — tampering is detected; a middlebox cannot silently rewrite bytes.
• Authentication — you are talking to the server that controls the certificate for that hostname.

What it does not provide: anonymity (the destination IP and the SNI hostname are visible), safety (an attacker’s site can have a perfectly valid certificate), or any protection once the bytes reach either endpoint.

The handshake, at a high level

TLS 1.3 (RFC 8446, 2018) simplified this considerably and removed every legacy cipher that had caused trouble. One round trip:

1. ClientHello — supported TLS versions, cipher suites, a key-share for guessed key-exchange groups, the SNI (the hostname being requested), and ALPN (which application protocols the client speaks: h2, http/1.1).
2. ServerHello — chosen cipher suite, the server’s key-share, chosen ALPN protocol. From here on, everything is encrypted.
3. Certificate + CertificateVerify — the server’s certificate chain, plus a signature proving it holds the matching private key.
4. Finished on both sides — a MAC over the whole handshake, so any tampering with earlier messages is detected.

Both sides derive the same symmetric session keys from the exchanged key-shares (an ephemeral Diffie-Hellman exchange). Because the private keys are ephemeral and discarded, capturing traffic today and stealing the server’s certificate key later does not decrypt it — that property is forward secrecy, and TLS 1.3 makes it mandatory.

TLS 1.2 needed two round trips and allowed non-forward-secret key exchange. It is still widely supported; anything below 1.2 is deprecated and should be off.

Certificates and the chain of trust

A certificate binds a public key to an identity (one or more hostnames), signed by a certificate authority. Verification walks a chain:

Leaf: CN=example.com          ← signed by
Intermediate: CN=DigiCert TLS RSA SHA256 2020 CA1  ← signed by
Root: CN=DigiCert Global Root CA  ← in your OS/browser trust store

The client checks, for every link: the signature is valid, the current time is inside the validity window, the certificate is not revoked, and — for the leaf — the requested hostname matches a name in the Subject Alternative Name extension. (The old CN field is ignored by modern browsers.) Roots are self-signed; they are trusted because they are pre-installed, not because anything vouches for them.

Public certificates are also logged to Certificate Transparency logs, which are publicly searchable — useful both for catching mis-issuance and, on the offensive side, for enumerating a target’s subdomains.

SNI

TLS happens before HTTP, so the server has no Host header yet — but it must pick a certificate. Server Name Indication solves this: the client puts the hostname in the ClientHello, in plaintext.

That plaintext hostname is why a network observer still knows which sites you visit over HTTPS. Encrypted Client Hello (ECH) addresses it and is rolling out, but is not yet universal.

Practical consequence: on a shared IP you must send the right SNI or you get the wrong certificate.

# Wrong: no SNI, server returns its default cert
openssl s_client -connect example.com:443 </dev/null

# Right: SNI set explicitly
openssl s_client -connect example.com:443 -servername example.com </dev/null

Inspecting TLS

# Full handshake trace, chain, protocol and cipher
openssl s_client -connect example.com:443 -servername example.com </dev/null

# Just the certificate, decoded
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -text

# Expiry dates only
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

# Subject Alternative Names — every hostname the cert covers
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

# Force a version to test support
openssl s_client -connect example.com:443 -servername example.com -tls1_2 </dev/null

In the output, look for Protocol : TLSv1.3, Cipher : TLS_AES_256_GCM_SHA384, and Verify return code: 0 (ok). A non-zero verify code names the exact failure — expired, self-signed, unable to get local issuer certificate (a missing intermediate, the single most common misconfiguration).

curl -v shows a condensed version of the same information, which is usually enough:

curl -v https://example.com/ -o /dev/null 2>&1 | grep -E '^\*'

Danger

curl -k (and --insecure) disables certificate verification entirely. It turns HTTPS into encryption without authentication, which stops nothing — an attacker in the middle just presents their own certificate. Use it to diagnose, never in anything that runs unattended.

HTTPS and HSTS

A user typing example.com gets http:// by default, and that first plaintext request can be hijacked. Two mitigations:

• HSTS — the Strict-Transport-Security response header tells the browser to use HTTPS for this host for the next N seconds, converting http:// links internally before any request goes out.
• The HSTS preload list — a list compiled into browsers, so even the very first visit is HTTPS. Requires includeSubDomains and preload in the header and a submission to hstspreload.org.

Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

See security headers for the rest of the set.

Key points

• A URL is scheme, optional userinfo, host, optional port, path, query and fragment — and the fragment never leaves the browser.
• Percent-encode individual values with encodeURIComponent, or build URLs with a URL library.
• DNS is a cached hierarchy: stub → recursive resolver → root → TLD → authoritative, with TTLs governing staleness.
• TCP costs one round trip before any data; TLS 1.3 costs one more. Reusing connections avoids both.
• TLS gives confidentiality, integrity and server authentication — not safety, not anonymity.
• Certificates are trusted via a chain up to a pre-installed root; SNI is sent in plaintext so the server can pick the right one.