"The Site Is Down!" — The 5-Command Linux Diagnostic Flow (dig, curl -v, ss, nc, tcpdump) to Pinpoint Any Outage
The Chaos of an Outage: Why Amateurs Guess and Engineers Diagnose
It is 14:15 on a Tuesday afternoon. Slack alerts erupt with messages in all-caps: “THE PRODUCTION SITE IS DOWN! CUSTOMERS CANNOT CHECK OUT!”
In inexperienced engineering teams, panic ensues. Developers begin frantically guessing:
- “Maybe we should restart Nginx!”
- “Did someone change the database password?”
- “Let’s reboot the entire virtual machine!”
- “Is AWS or Cloudflare having a global outage?”
Rebooting servers and blindly restarting services is not engineering; it is superstition. Guessing wastes precious minutes, wipes volatile kernel socket states and connection queues that hold the root cause, and frequently compounds the outage.
THE 5-STEP DETERMINISTIC DIAGNOSTIC PIPELINE
- “THE SITE IS DOWN!”
- STEP 1: DNS LAYER: dig +trace api.example.com Did the domain resolve to the right IP?
- STEP 2: APP / TLS LAYER: curl -Iv https://api.example.com/health
- Does TLS complete? What HTTP code is returned?
- STEP 3: LOCAL SOCKETS: ss -tulpn | grep -E “80|443”
- Is Nginx actually listening on the port?
- STEP 4: PORT FIREWALL: nc -zv 192.168.1.50 443
- Can traffic traverse internal iptables/VPC?
- STEP 5: PACKET CAPTURE: tcpdump -nn -i eth0 port 443
- Are TCP SYN packets arriving and answering?
When an incident occurs, you need a deterministic, linear diagnostic flow that isolates the failure domain within 180 seconds.
Through our managed hosting and proactive maintenance services, our engineers rely on five battle-tested Linux tools on Debian GNU/Linux to pinpoint the root cause of any outage with mathematical certainty.
Step 1: DNS Resolution Verification with dig
The majority of “web server outages” are not server outages at all—they are DNS failures. A domain registrar expired, an NS record was stripped, or a DNSSEC signature lapsed.
Execute an Authoritative Trace
Do not rely on your local laptop’s cached DNS. Query the entire DNS delegation tree starting from the ICANN root zone using dig:
# Trace the complete delegation chain from root servers down to your nameservers
dig +trace api.example.com
SAMPLE DIG DELEGATION OUTPUT
- . 518400 IN NS a.root-servers.net. com. example.com.
- 172800 IN NS a.gtld-servers.net.: 172800 IN NS ns1.cloudflare.com. api.example.com. 300 IN A 203.0.113.80
- ;; Received 240 bytes from 173.245.58.100#53(ns1.cloudflare.com) in 14 ms
What to Look For:
- If
digreturnsSERVFAIL, your DNSSEC signatures are expired or broken. - If
digreturnsNXDOMAIN, your DNS zone was deleted or misspelled. - If
digreturns the wrong IP address, an automated CI/CD pipeline or registrar pointed traffic to a decommissioned server.
Step 2: Protocol Handshake & HTTP Inspection with curl -v
Once you have verified that DNS resolves to the correct IP address, test the HTTP and TLS layers using verbose curl:
# Verbose TLS handshake and header inspection, ignoring client cache
curl -Iv https://api.example.com/healthz
CURL VERBOSE DIAGNOSTIC BREAKDOWN
- Connected to api.example.com (203.0.113.80) port 443 (#0) → [TCP OK]
- ALPN: server accepted h2
- HTTP/2 OK
- SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 → [TLS OK]
- Server certificate: CN=api.example.com, valid to 2026-12-15 → [Cert OK]
-
GET /healthz HTTP/2
-
Host: api.example.com < HTTP/2 502 Bad Gateway
- CULPRIT!
- < server: nginx/1.24.0
Interpreting the Diagnosis:
Look at the output above:
- The TCP handshake succeeded (
Connected to port 443). - The TLS 1.3 handshake succeeded with a valid certificate.
- Nginx responded with
502 Bad Gateway.
The Diagnosis: The server is online, the network is healthy, and Nginx is running. The outage is caused by the backend upstream application (Node.js, Python, or PHP-FPM) having crashed or refusing connections on port 3000. Do not touch Nginx; investigate the backend process!
Step 3: Kernel Socket Inspection with ss (Socket Statistics)
SSH into the target server. Is the application daemon actually running, or did it exit due to an unhandled exception?
Do not use the deprecated, slow netstat command. Use ss:
# List all TCP listening sockets with process IDs and numerical ports
ss -tulpn
SAMPLE SS SOCKET STATISTICS OUTPUT
- Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp LISTEN 0 511 0.0.0.0:80 tcp LISTEN 0 511 0.0.0.0:443
- 0.0.0.0:* users:((“nginx”,pid=1420,fd=6)): 0.0.0.0:* users:((“nginx”,pid=1420,fd=7)) tcp LISTEN 128 128 127.0.0.1:3000 0.0.0.0:* users:((“node”,pid=89412,fd=19))
Analyzing the Columns:
Local Address:Port: Notice127.0.0.1:3000. The Node.js application is bound only to the localhost loopback adapter. If Nginx is trying to reverse-proxy to192.168.1.50:3000(the LAN IP), the connection will be rejected!Recv-Q(Receive Queue): IfRecv-QequalsSend-Q(e.g.,128 128), the application’s socket backlog is completely saturated. The process is deadlocked or stuck in an infinite loop and cannot accept new connections!
Step 4: Layer 4 Port Reachability with netcat (nc)
What if the process is running on the backend host, but the web proxy cannot reach it across the internal network?
Test Layer 4 reachability with nc (Netcat):
# Test TCP connectivity with a 2-second timeout without sending data
nc -zv -w 2 10.0.0.15 3000
Reading Netcat Results:
Connection to 10.0.0.15 3000 port [tcp/*] succeeded!
The network route is open, the port is listening, and traffic flows freely.nc: connect to 10.0.0.15 port 3000 (tcp) failed: Connection refused
The packet reached the target machine, but no daemon is listening on that port, or the daemon is bound to127.0.0.1.nc: connect to 10.0.0.15 port 3000 (tcp) timed out
Firewall issue! A firewall (UFW,nftables, AWS Security Group, or cloud VPC rule) is silently dropping SYN packets.
Step 5: The Ultimate Truth: Raw Packet Capture with tcpdump
When all else fails, you must inspect the raw wire frames using tcpdump. Wire frames do not lie.
# Capture 20 packets on eth0 for port 443 with numerical output
tcpdump -nn -c 20 -i eth0 port 443
TCP THREE-WAY HANDSHAKE ON THE WIRE
- 14:20:01.102 IP 198.51.100.12.54210 > 203.0.113.80.443: Flags [S], seq 1000
- 14:20:01.102 IP 203.0.113.80.443 > 198.51.100.12.54210: Flags [S.], seq 5000, ack 1001
- 14:20:01.144 IP 198.51.100.12.54210 > 203.0.113.80.443: Flags [.], ack 5001
The Three Diagnostic Scenarios:
- Scenario A (Normal): You see
Flags [S](SYN) followed immediately byFlags [S.](SYN-ACK) andFlags [.](ACK). The three-way handshake completed in 42ms. - Scenario B (Server Resets): You see
Flags [S]from the client, followed immediately byFlags [R.](RST/ACK) from the server. The kernel actively rejected the connection because the listening port is closed. - Scenario C (Asymmetric Routing / Black Hole): You see repeated
Flags [S]packets retransmitted by the client every 3 seconds, but the server never transmits a SYN-ACK. Either the server’s egress routing table has the wrong default gateway, or an upstream firewall dropped the packet.
The Outage Reference Card: 3-Minute Cheat Sheet
Keep this diagnostic decision matrix pinned in your incident response channel:
| Diagnostic Output | Root Cause | Immediate Action |
|---|---|---|
dig returns SERVFAIL |
DNSSEC validation failure | Check DNS registrar DS records & TTL |
curl hangs on TLS handshake |
MTU / PMTUD black hole | Clamp TCP MSS or check packet loss with mtr |
curl returns 502 Bad Gateway |
Upstream backend service down | Check systemctl status <app> |
ss shows Recv-Q full |
App event loop / threads blocked | Restart app worker / check CPU profile |
nc returns Connection timed out |
Firewall dropping packets | Inspect nft list ruleset or Cloud VPC rules |
tcpdump shows RST packets |
Port closed on local host | Verify listening interface (0.0.0.0 vs 127.0.0.1) |
By executing this disciplined 5-command protocol, your engineering team transforms high-stress outages into calm, methodical investigations. Rooted in our technical consulting and anti-over-engineering principles, you master infrastructure by understanding how the Linux kernel and network protocols communicate. Explore automated incident triage scripts on GitHub to equip your team with standardized diagnostic workflows today.
Alongside our operational work at Webxtek Studio, we apply these exact architectural standards across specialized sectors including devops & cloud architecture specialists and cloud computing & iaas providers. Whether you are modernizing core workflows through managed hosting for websites and apps and technical consulting before you build, connecting sector-specific requirements to disciplined execution is what turns technical theory into compounding commercial advantage.
[+] [ INDEX_DIRECTORY ] 21 topics ▼
- The Chaos of an Outage: Why Amateurs Guess and Engineers Diagnose
- Step 1: DNS Resolution Verification with dig
- Execute an Authoritative Trace
- What to Look For:
- Step 2: Protocol Handshake & HTTP Inspection with curl -v
- Interpreting the Diagnosis:
- Step 3: Kernel Socket Inspection with ss (Socket Statistics)
- Analyzing the Columns:
- Step 4: Layer 4 Port Reachability with netcat (nc)
- Reading Netcat Results:
- Step 5: The Ultimate Truth: Raw Packet Capture with tcpdump
- The Three Diagnostic Scenarios:
- The Outage Reference Card: 3-Minute Cheat Sheet
- Author
- Frequently Asked Questions
- What is the optimal systematic order for Linux network troubleshooting?
- Why should you use ss instead of the legacy netstat command?
- How does curl -v distinguish between DNS, connection, and TLS handshake latency?
- What is the safest syntax for capturing production traffic with tcpdump without freezing the terminal?
- Related Nodes
- Path Sequence
Frequently Asked Questions
What is the optimal systematic order for Linux network troubleshooting?
Troubleshooting must progress strictly from Layer 7 down to Layer 3: 1. DNS Resolution (`dig +trace`), 2. Application Layer HTTP/TLS Handshake (`curl -v`), 3. Local Kernel Socket Listeners (`ss -tulpn`), 4. Layer 4 Transport Firewall Reachability (`nc -zv`), and 5. Raw Packet Inspection (`tcpdump -nn`).
Why should you use ss instead of the legacy netstat command?
The legacy `netstat` command parses `/proc/net/tcp` line-by-line in user space, causing massive CPU spikes and taking minutes to run on servers with tens of thousands of active sockets. The modern `ss` (Socket Statistics) command interacts directly with the Linux kernel's `sock_diag` netlink interface, dumping socket states instantaneously with minimal overhead.
How does curl -v distinguish between DNS, connection, and TLS handshake latency?
By pairing `curl -v` with write-out timing variables (`curl -w "@curl-format.txt"`), curl breaks down total request duration into individual mathematical components: `time_namelookup`, `time_connect` (TCP handshake), `time_appconnect` (TLS negotiation), and `time_starttransfer` (server TTFB).
What is the safest syntax for capturing production traffic with tcpdump without freezing the terminal?
Always use `tcpdump -nn -c 100 -i eth0 port 443`. The `-nn` flags disable DNS name resolution and port service translation (preventing recursive DNS lookups from locking up the tool), `-c 100` stops capture after 100 packets, and explicit port/interface filters prevent terminal buffer flooding.
[ RELATED_NODES ]
> START_PROJECT
Need a website that earns trust, ranks in search, and gives your business a stronger digital presence? Start the conversation here.