Payment Terminals Offline During Peak Hours Due to Broadcast Storm Saturation or High Bandwidth Utilization
Environment: Restaurant Network (Router + Managed Switch + POS + Guest Wi-Fi + IoT Devices)
1. Purpose
This document defines the troubleshooting and remediation process for incidents where payment terminals go offline during peak business hours due to broadcast storm saturation or High Bandwidth Utilization
This is a critical business-impacting event because it affects payment processing during high revenue periods.
2. Scope
Applies to:
POS terminals
Payment processing devices
Managed Layer 2 switches
Guest Wi-Fi networks
IoT devices (cameras, printers, kiosks)
3. Background – How does Broadcast Storm or High Bandwidth Utilization impact the business continuity
A broadcast storm is a condition where an excessive number of broadcast packets floods the network, overwhelming network devices and degrading performance.
Broadcast storms occur when too many broadcasts, multicasts, or unknown unicast frames are generated in a short period of time in a looped environment or heavy load/ bandwidth bottleneck due to large no of users are connected.
Examples of broadcast traffic include ARP requests, DHCP discovery messages, unknown unicast flooding, and misclassified multicast.
High bandwidth utilization creates network bottlenecks and disrupts operational communication.
5. What Happens Technically
When broadcast levels spike:
ARP responses are delayed.
DHCP requests are delayed or dropped.
Payment terminals cannot resolve gateway MAC.
Internet browsing may still partially function because:
Some sessions are already established.
Not all VLANs equally impacted.
Partial packet loss tolerated by some applications.
When bandwidth utilization spike:
Network links become saturated.
Packet queues on switches/routers begin to fill.
Packets are delayed due to increased queuing time.
Packet drops occur when buffers is full.
Latency and jitter increase significantly.
Critical application traffic competes with non-critical traffic.
6. Typical Symptoms
Staff Reports
POS freezing during rush.
Payments timing out.
Internet seems slow but not fully down.
Happens during evening or peak hours.
Technical Indicators
High broadcast traffic counters on device interface
All port LEDs blinking heavily.
Intermittent packet loss.
Frequent connection and disconnections on client list
High bandwidth usages on portal
7. Business Impact
Payment processing failure during peak hours.
Customer delays.
Lost transactions.
Manual fallback procedures.
Revenue loss.
Severity Level: Critical during business hours
8. Common Root Causes
Layer 2 loop (most severe).
Guest Wi-Fi VLAN overloaded.
IoT device flooding ARP requests.
Misconfigured device generating multicast storms.
Large flat network (no VLAN segmentation).
High Bandwidth Utilization
9. Detailed Troubleshooting Procedure
Step 1 – Confirm Broadcast Saturation
Log into switch.
Check:
Broadcast packet counters
CPU utilization
Interface statistics
Error counters
Check any multiple events in PCC portal (DHCP Lease, Port UP/Down, Client association or disassociation)
Step 2 – Identify Storm Source Port
Look for:
Port with abnormally high broadcast output.
Port with excessive unknown unicast traffic.
Port counters increasing rapidly.
Port with A CRC (Cyclic Redundancy Check) counter increasing
Disconnect suspected port temporarily.
If storm stops → root cause isolated.
Step 3 – Check for Layer 2 Loop
Signs of loop:
MAC address flapping.
Same MAC appearing on multiple ports.
Switch logs showing topology change events.
STP logs or port going in block mode again & again
If loop detected:
Immediately disconnect one of redundant links.
Step 4 – Evaluate VLAN Segmentation
Check whether:
Guest Wi-Fi shares VLAN with POS.
IoT devices share broadcast domain with payment systems.
Guest Users are consuming high bandwidth.
Flat networks are highly vulnerable to broadcast storms.
10. Resolution Procedures
Scenario A – Layer 2 Loop
Disconnect redundant cable.
Enable STP (Spanning Tree Protocol).
Enable BPDU Guard on access ports.
STP prevents loops automatically.
Scenario B – High Broadcast from Guest Devices
Segment Guest Wi-Fi into separate VLAN.
Apply rate limiting via QoS
Enable storm control on Interface
Scenario C – IoT Device Flooding
Identify offending device.
Check for devices which are consuming more bandwidth
Update firmware.
Replace malfunctioning unit.
Move IoT devices to isolated VLAN.
Scenario D – No Storm Control Configured
Enable Storm Control:
Limit broadcast percentage per port.
Set threshold (e.g., 5–10%).
Configure automatic shutdown if exceeded.
Storm control prevents single device from overwhelming network.
Scenario E – Identify clients with High bandwidth utilization
Check PCC portal client page to identify current clients with high bandwidth utilization
Check and identify clients consuming excessive bandwidth
Identify business (POS, Hand Handles) or non-business clients (Guest users)
Block the non-business clients and check why if any business client consuming high bandwidth
11. Validation After Fix
Confirm:
Broadcast traffic normalized.
CPU stable under load.
No packet loss.
POS processes payments reliably.
No storm recurrence during next peak hour.
No High Bandwidth Utilization
Monitor during next rush period.
12. Preventive Measures
Enable STP on all switches.
Enable BPDU Guard.
Enable storm control.
Segment POS, Guest, and IoT VLANs.
Regularly audit for cable loops.
Use managed switches (not unmanaged switches/hubs).
Avoid daisy-chaining small switches.
Apply QoS to avoid High bandwidth utilization