VLAN, Inter-VLAN Routing, and Firewall ACL Misconfiguration — Network Segmentation Failure

Indicators

• Hosts in separate VLANs cannot communicate despite inter-VLAN routing being configured
• Unexpected packet drops between VLAN segments with no corresponding firewall deny log entries
• Traffic from one VLAN appears on a VLAN it should not have access to (VLAN leakage)
• Routing table on Layer 3 switch or router is missing directly connected or static routes for one or more VLANs
• Firewall rule hit counters show zero hits on rules expected to permit inter-VLAN traffic (Cisco ASA: `show access-list <name>`; iptables: `iptables -L -v -n`)
• Hosts receive IP addresses from the wrong VLAN's DHCP scope, indicating incorrect access port VLAN assignment or trunk tagging error
• Ping or traceroute between VLAN gateways fails at the first hop
• CDP/LLDP warnings about native VLAN mismatch on trunk links

Likely causes

• Switch port configured as access port instead of trunk port, preventing multiple VLAN tags from being carried across an inter-switch link
• VLAN ID not added to the allowed VLAN list on a trunk link — frames for that VLAN are silently dropped at the trunk boundary
• Missing or shutdown VLAN interface (SVI) on Layer 3 switch — no gateway exists for the VLAN subnet, preventing all inter-VLAN routing
• Incorrect or absent static routes on the router or firewall for VLAN subnets — traffic is dropped or sent to the wrong next-hop
• Firewall rules referencing wrong source/destination zone, interface, or subnet — implicit deny matches legitimate inter-VLAN traffic
• Native VLAN mismatch on trunk ports between switches — untagged frames land in the wrong VLAN, causing misdelivery or loops
• ACL applied in the wrong direction (inbound vs outbound) on a routed VLAN interface — blocks traffic that should be permitted
• VLAN ID mismatch between switches due to VTP misconfiguration or manual error (e.g., VLAN 10 on one switch not propagated to an adjacent switch)

Diagnostic steps

1. Identify the affected VLANs and verify VLAN existence on every switch in the traffic path. Cisco IOS: `show vlan brief` — confirm expected VLAN IDs are present and in 'active' state, not 'act/unsup' or absent. Open vSwitch: `ovs-vsctl show`. Linux bridge: `bridge vlan show`.
   Confirms whether the VLAN is defined and active on each switch, ruling out a missing VLAN database entry as the root cause before investigating trunk or routing issues.
2. Check trunk port configuration and allowed VLAN lists on all inter-switch links. Cisco IOS: `show interfaces trunk`. Verify the affected VLAN ID appears in the 'VLANs allowed and active in management domain' column for each trunk interface — not just 'VLANs allowed on trunk', as a VLAN can be allowed but inactive if it is not in the local VLAN database.
   Determines whether the VLAN is permitted to traverse inter-switch trunk links; a missing entry causes all frames for that VLAN to be dropped silently at the trunk boundary.
3. Verify Layer 3 VLAN interfaces (SVIs) or routed sub-interfaces exist and are in up/up state. Cisco IOS: `show ip interface brief` — locate `Vlan<ID>` entries; both 'Status' and 'Protocol' columns must show 'up'. Confirm the assigned IP address matches the intended default gateway for the VLAN subnet. If the SVI is administratively down, issue `no shutdown` under `interface vlan <id>` and recheck.
   Confirms that the routing device has an active, correctly addressed gateway interface for each VLAN; a missing or shutdown SVI prevents all inter-VLAN routing for that segment.
4. Review the routing table on the Layer 3 switch, router, or firewall for all VLAN subnets. Cisco IOS: `show ip route`. Linux: `ip route show`. Confirm directly connected routes appear for each SVI subnet (marked 'C') and that no VLAN subnet route is missing or pointing to an incorrect next-hop. Flag any VLAN subnet absent from the table.
   Identifies missing or incorrect routes that would cause traffic to be dropped at the routing device or forwarded to the wrong next-hop, manifesting as one-way or no connectivity.
5. Inspect firewall and ACL rules for rules that explicitly deny or fail to permit inter-VLAN traffic. Review rule order, source/destination zones or interfaces, and subnet definitions carefully. Check hit counters: Cisco ASA: `show access-list <name>`, looking for zero-hit permit rules and unexpectedly high-hit deny rules. Linux iptables: `iptables -L -v -n`. Identify shadowed rules (a broader deny before a specific permit) and rules applied to the wrong interface or direction.
   Determines whether a firewall deny rule or an absent permit rule is blocking legitimate inter-VLAN traffic, and whether rules are being evaluated in the intended order — a frequent silent failure mode.
6. Use packet capture or traffic mirroring to observe actual frames at the VLAN boundary. Cisco IOS: configure a SPAN session (`monitor session 1 source interface <src>`, `monitor session 1 destination interface <dst>`) — avoid `debug ip packet` on production hardware under load. Linux/firewall: `tcpdump -i <interface> -nn 'host <src-ip> and host <dst-ip>'`. Confirm whether packets are arriving at the routing or firewall device, whether they are being forwarded, and whether ICMP unreachables or TCP RSTs are being returned.
   Provides ground-truth visibility into whether traffic is reaching the inter-VLAN routing point and what is happening to it, conclusively distinguishing a Layer 2 forwarding problem from a Layer 3 routing miss or a firewall policy block.

Resolution path

• 1. Correct missing or misconfigured VLAN definitions: On each switch where the VLAN is absent, enter global config and create it: `vlan <id>` followed by `name <descriptive-name>`. Verify it transitions to 'active' in `show vlan brief` and is not in 'act/unsup' state.
• 2. Fix trunk port allowed VLAN lists: On each trunk interface missing the VLAN, add it without replacing the existing list — Cisco IOS: `switchport trunk allowed vlan add <vlan-id>`. Never use `switchport trunk allowed vlan <vlan-id>` alone as this replaces the entire list. Verify with `show interfaces trunk` that the VLAN now appears under 'VLANs allowed and active in management domain'.
• 3. Create or repair the Layer 3 SVI: On the Cisco IOS Layer 3 switch: `interface vlan <id>` → `ip address <gateway-ip> <subnet-mask>` → `no shutdown`. Verify with `show ip interface brief` that the interface shows 'up' / 'up' and the IP address is correct.
• 4. Add or correct static or connected routes: If a separate router or firewall is the inter-VLAN routing point and lacks a route, add: `ip route <network> <mask> <next-hop-ip>`. If a dynamic routing protocol (OSPF, EIGRP) is in use, verify the VLAN SVI network is included in the redistribution or network statements.
• 5. Correct firewall or ACL rules: Insert a specific permit rule for the required inter-VLAN traffic above any deny rules, specifying the correct source zone/interface and subnet, destination zone/interface and subnet, and required ports/protocols. Apply ACLs in the correct direction (`ip access-group <name> in` vs `out`) on the correct interface. Remove shadowed, duplicate, or conflicting rules. After change, verify hit counters increment on the new permit rule.
• 6. Resolve native VLAN mismatches on trunk links: Ensure both ends of each trunk have identical native VLAN configuration — Cisco IOS: `switchport trunk native vlan <id>` on both ends. Native VLAN mismatches cause CDP warnings and untagged frame misdelivery; if no native VLAN tagging is required, set native VLAN to an unused VLAN ID on all trunks as a security measure.
• 7. Save configuration: Cisco IOS: `copy running-config startup-config` (or `write memory`). Retest connectivity between hosts across all affected VLANs using ping and traceroute.

Prevention

• Implement a formal VLAN change management process: all VLAN, trunk, and firewall rule changes must be peer-reviewed, tested in a staging environment or maintenance window, and accompanied by a documented rollback plan before production deployment.
• Use network configuration management tools (Ansible, Cisco DNA Center, NetBox) to maintain authoritative VLAN and ACL definitions as infrastructure-as-code, enabling automated diff-checking, drift detection, and one-command rollback.
• Enable and monitor CDP/LLDP native VLAN mismatch warnings on all trunk ports; configure syslog and SNMP traps to alert on interface state changes, VLAN topology events, and STP topology changes.
• Regularly audit firewall and ACL rule bases to identify shadowed rules (specific permit after broad deny), duplicate rules, zero-hit rules, and overly permissive entries; remove unused rules and document business justification for every inter-VLAN permit.
• Apply the principle of least privilege to inter-VLAN firewall policy: default-deny between all VLANs and explicitly permit only the required ports, protocols, and source/destination subnet pairs.
• Automate configuration backup after every change using RANCID, Oxidized, or vendor-native tools; verify backups are retrievable and test restoration procedures quarterly.

Tools

• show vlan brief (Cisco IOS — list active VLANs on a switch)
• show interfaces trunk (Cisco IOS — verify trunk links and allowed/active VLAN lists)
• show ip interface brief (Cisco IOS — confirm SVI/routed interface up/up state and IP address)
• show ip route (Cisco IOS — inspect routing table for VLAN subnets)
• show access-list <name> (Cisco IOS/ASA — review ACL rules and hit counters)
• iptables -L -v -n (Linux — review firewall rules with packet counters)
• tcpdump (Linux/Unix — packet capture for traffic verification at VLAN boundary)
• ovs-vsctl show / bridge vlan show (Open vSwitch / Linux bridge — VLAN and port inspection)
• SPAN/RSPAN (Cisco — traffic mirroring to analysis port without impacting forwarding)
• ping / traceroute (end-to-end connectivity test and hop-by-hop path verification)
• Wireshark (GUI packet analysis of SPAN or tcpdump captures)

References

• Cisco IOS — Configuring VLANs and Trunking (Catalyst 9300)
• Cisco IOS — Inter-VLAN Routing Configuration Guide
• RFC 8214 — Virtual Private Wire Service Support in Ethernet VPN (EVPN) — VLAN tagging reference