Firewall Policy Validation

VLANs and Security Boundaries

A VLAN by itself is not a firewall, and I think that distinction gets blurred way too often. At Layer 2 a VLAN separates broadcast domains, while VLAN tagging allows that separation to continue across trunk links between switches and other network devices. None of that automatically means two IP networks cannot communicate with each other.

Once multiple VLANs have gateways on a router or Layer 3 switch, that device can route traffic between them unless some other policy prevents it. So I could build something that looks perfectly reasonable on paper:

VLAN 10 - Management
VLAN 20 - IoT

and still have devices in VLAN 20 establishing connections straight into management services on VLAN 10 if I never actually created and enforced the policy between them.

The VLAN names make the networks look separated, but routing and firewall policy are what determine whether that separation actually becomes an access boundary. Layer 2 segmentation is part of the design. It is not the entire security control.

Testing the Actual Service

One of the most common firewall tests is also one of the least useful if it is the only thing being tested:

ping 192.0.2.10

If ping fails, it is really easy to immediately jump to the conclusion that the host is blocked. The problem is that ping is testing ICMP Echo and that is really all the result tells me.

A firewall could be dropping ICMP while TCP/443 is completely reachable, or the destination operating system might simply ignore ICMP while an SSH server, web server, or some other application is listening normally. So if I care about whether TCP/443 is reachable, I test TCP/443 instead of asking an unrelated protocol to answer the question for me.

For a basic TCP check:

nc -zv -w 2 192.0.2.10 443

If it is actually an HTTPS service, I would rather go one step further and test the service itself:

curl -kv --connect-timeout 2 https://192.0.2.10/

That gives me a much more useful observation because now the test actually resembles the traffic I care about. Even then, though, a successful or failed connection does not automatically explain why it behaved that way.

That is where the next part matters.

Packet Capture and Attribution

When traffic behaves differently than I expected, packet capture is normally one of the first places I go because guessing at a network problem gets old pretty quickly. Depending on what I am testing, I might capture from the source, the gateway, the destination, or several of them at the same time.

For example:

tcpdump -n -i eth1.110 host 192.0.2.50 and port 443

and then on the destination:

tcpdump -n -i eth0 src 192.0.2.50 and port 443

Those two captures can tell me much more than simply knowing that the connection timed out. I may be able to prove that the SYN actually left the source, that it appeared on a particular interface, that it reached the destination, or that the destination answered and the reply disappeared somewhere on the way back.

The important part for me now is understanding exactly what a capture proves from the point where I collected it. Seeing a packet on one interface does not automatically prove that the packet reached another interface, and seeing traffic disappear somewhere along a route does not automatically mean I can point at a particular firewall rule and say that rule definitely caused it.

That was something I used to be a little less strict about.

Now I want the observation and the attribution whenever I can reasonably get both.

IPv4 and IPv6

Dual-stack networking creates another really easy place for configuration assumptions to fall apart because IPv4 and IPv6 can have completely different policy states even though the person using the network only sees one application.

I can have a strict IPv4 rule between two networks while the equivalent IPv6 path is still allowed. If the destination publishes both A and AAAA records, modern clients can attempt both protocols and select a working path, which means the application may still connect over IPv6 even while the IPv4 firewall policy is doing exactly what I intended.

For example:

IPv4
Subnet A -> Subnet B
BLOCKED

IPv6
Subnet A -> Subnet B
ALLOWED

From the user's perspective the application works. From the firewall administrator's perspective the IPv4 deny rule also works. Both statements can be true at the same time, which is exactly why I do not consider testing the IPv4 path proof of what is happening over IPv6.

If both protocols exist on the network, then both protocols need to be considered during testing.

Rule Matching

I ran directly into this while rebuilding my own network.

I had a policy in UniFi that looked correct in the management interface, but the traffic was not matching the rule the way I expected it to. At that point the counters were much more useful to me than continuing to stare at the GUI because the counters showed that the traffic was actually reaching a later drop rule instead of the allow rule I thought should be matching it.

During troubleshooting I replaced the network-object match with a narrow literal IPv6 CIDR:

Source:      2001:db8:b1f8::/64
Destination: 2001:db8:b1f1::10
Port:        443
Action:      Accept

After that change, the intended traffic started matching the path I expected.

That result told me something useful, but I also tried not to stretch the result further than it could go. It showed that the original traffic was not being evaluated the way the management interface had led me to expect, and it gave me somewhere much more specific to continue troubleshooting.

It did not automatically tell me exactly why UniFi produced that behavior in the first place.

That difference matters because finding the location of a problem and proving the actual root cause are not always the same thing.

Layered Validation

When I validate an access boundary now, I normally look at several layers instead of treating one firewall rule as the entire security design.

Gateway Policy

First I test whether traffic crossing between networks is actually allowed or denied the way the policy says it should be. This is where inter-VLAN rules and other routing boundaries become something I can test instead of something I only read from a configuration screen.

Host Firewall

I also want to know what happens if the upstream policy is ever wrong. A host firewall such as UFW or Windows Defender Firewall gives me another control point, and I would rather know that local access policy is doing what I intended before I actually need it.

Listener State

Another thing I have become much more interested in is what the application is actually bound to.

On Linux I might start with:

ss -lntp

A service listening on:

127.0.0.1:8080
[::1]:8080

is in a completely different state from one listening on:

0.0.0.0:8080
[::]:8080

even if a firewall makes a remote connection fail in both cases. If something was only supposed to be reachable locally, I would much rather have the service actually listening locally instead of listening everywhere and depending entirely on another control to hide it.

Application Behavior

Finally, if the application is supposed to work, I test the actual application.

If I am validating HTTPS, I make an HTTPS request. If I am validating SSH, I test SSH. If I am validating DNS, I send a DNS request.

The closer the test gets to the real traffic the system is supposed to handle, the more useful the result normally becomes.

Positive and Negative Testing

Testing that something works is the easy side of validation. The other side is proving that the things that should not work actually fail.

For example:

Authorized management host -> HTTPS application
Expected: reachable

Then I might test the opposite side of the same boundary:

Untrusted network -> management SSH
Expected: denied

or:

DMZ -> management interface
Expected: denied

A failed negative test doesn't automatically prove the security control caused the failure and if the destination service is dead, the connection fails.

Just like if the route is broken the connection fails, if nothing is listening on the port the connection fails, or if the firewall drops it the connection also fails. Those can all look very similar from the source, so when the distinction actually matters I try to establish target liveness and use additional evidence such as packet captures, counters, logs, listener state, or another useful observation point to figure out why the test failed.

Sometimes I still cannot prove it.

In that case I would rather leave the result inconclusive than take a timeout and turn it into firewall evidence just because that happens to be the answer I expected.

Evidence

This is probably the biggest thing that changed in how I look at firewall validation because I no longer think of “evidence” as one single type of result.

A firewall rule in a management interface tells me what I configured. A connection timeout tells me that the connection did not complete. A packet capture tells me that a packet was observed at a particular point, while a listener check can tell me whether the application is actually listening on a particular interface or port.

A policy counter can add another piece by showing that a specific rule matched traffic while I was performing the test.

None of those by themselves automatically answer every question I have, but when I can line several of them up against a destination that I already proved was alive, the conclusion becomes much stronger.

For example, if I know a management service is listening and reachable from an authorized system, I send a SYN from an untrusted network, I see that SYN leave the source, I see a matching deny counter increment, and I see no packet arrive at the destination interface, I have a much stronger argument than:

That is closer to the standard I want to use now.

Conclusion

A firewall rule is still important. Obviously.

But the existence of the rule is not the part I consider proof anymore because all it really tells me is what the intended state should be.

Traffic testing tells me what I actually observed, while packet captures, counters, logs, listener state, and target-side evidence can help me figure out why I observed it.

That difference can sound pretty small when everything is working exactly the way I expect, but it becomes much more important when the network suddenly does something the dashboard says it should not be doing.

The idea I keep coming back to is simple: