Firewall / network change broke connectivity
Recent firewall or network change has caused an outage. Roll back fast, then diagnose what was wrong about the change.
Indicators
- Outage starts immediately after a change window
- Some traffic works, other traffic doesn't (rule-specific)
- Symmetric NAT / asymmetric routing post-change
- Recent firmware upgrade preceded loss of connectivity
Likely causes
- New rule placed too low in policy order (implicit deny matched first)
- Object-group reference now pointing at wrong subnet
- NAT changed and broke an inbound service
- Firmware upgrade removed deprecated config / changed defaults
- VLAN trunk allowed-list shortened, removing traffic
Diagnostic steps
-
Confirm change happened — review change log, ask who touched what
-
Have a tested rollback before any further change. On Fortinet: execute backup config / restore. On SonicWall: import previous .exp. On Meraki: there is no rollback — use config-versioning or revert via dashboard history
-
If rollback restores service: diff the configs to identify the bad delta
-
If rollback isn't safe: identify the failing flow by packet capture from each side, compare to policy
-
Apply minimal corrected change, test, document
Resolution path
- Roll back to last-known-good if available
- If not, identify and correct the specific bad rule / NAT / VLAN
- Re-test all critical flows post-change
- Update documentation and monitoring
Prevention
- Out-of-band access for every firewall (console + IPMI)
- Config backup before AND after every change
- Change windows with rollback timer (auto-revert if not confirmed in 10min)
- Documented baseline config per device, version-controlled
Tools
- Vendor CLI / GUI for config diff / restore
- Wireshark on both sides for asymmetric routing
- Cisco ASA: packet-tracer; FortiGate: diagnose debug flow
- Out-of-band management (console cable, IPMI) — assume in-band can fail
References
- Vendor change management guidance
- Cisco / Fortinet config-diff / restore guides