“Cannot reach the server” can describe dozens of failures: a down interface, wrong address, missing default route, incorrect DNS, a service bound only to localhost, dropped packets, wrong TLS SNI, or a stalled upstream behind a proxy. Running random commands usually adds noise.
A better method starts at lower layers, tests one hypothesis at a time, and compares observations from both client and server. This guide uses common Linux tools: ip, ss, ping, dig, curl, openssl, traceroute, and tcpdump.
Before changing anything: define the blast radius
Record who is affected, which service, when it began, and what changed. Determine whether it affects one or all clients, one port or all Internet access, IPv4 or IPv6, a host/VM/container, and external traffic or localhost too.
Do not restart everything immediately. Restarts erase useful state, logs, and connections. Capture command output, timestamps, and configuration before modification.
1. Check interfaces and physical link
ip -br link
ip -br address
ip -s link show dev eth0
UP means administratively enabled; LOWER_UP means the kernel sees carrier. Without it, inspect cable, switch port, virtual NIC, hypervisor, or wireless association.
ip -s link exposes packet errors and drops. Counters rising quickly can indicate driver, duplex, MTU, buffering, or congestion problems. Compare two samples rather than one lifetime total.
sudo ethtool eth0
sudo ethtool -S eth0
On physical servers, ethtool provides carrier, speed, duplex, and driver statistics. Some values are less meaningful on virtual interfaces.
2. Verify IP addressing
ip address show
ip -4 address show dev eth0
ip -6 address show dev eth0
Confirm the address, prefix, interface, and tentative/deprecated status. A wrong prefix may make the host ARP for a remote destination instead of using its gateway.
networkctl status eth0
journalctl -u systemd-networkd --since '-15 min'
nmcli device show eth0
Use the commands matching the active network manager. Ubuntu Server commonly persists configuration through Netplan, backed by systemd-networkd or NetworkManager.
3. Inspect the route the kernel will use
ip route show
ip -6 route show
ip route get 1.1.1.1
ip route get 203.0.113.50 from 192.0.2.10
ip route get shows the selected interface, gateway, and source address for a specific destination. Multi-homed systems may also use policy routing:
ip rule show
ip route show table all
Look for overlapping routes, metrics, default gateways, and source-based rules. Asymmetric routing can bring a request through one interface and send the response through another, where a firewall or upstream drops it.
4. Inspect neighbors and the gateway
ip neigh show
ping -c 3 192.0.2.1
arping -I eth0 -c 3 192.0.2.1
FAILED or INCOMPLETE neighbor state means the host cannot resolve the MAC address on the local link. Causes include a wrong VLAN, prefix, gateway, hypervisor security group, or different broadcast domain.
Blocked ping does not prove a host is down. Continue with the actual TCP port. But if the local gateway gives no ARP response, the problem is below DNS and TLS.
5. Separate networking from DNS
resolvectl status
resolvectl query api.example.com
dig api.example.com A
dig api.example.com AAAA
dig @1.1.1.1 api.example.com A
getent ahosts api.example.com
dig queries DNS directly, while getent follows the Name Service Switch used by many applications, including hosts files and other resolvers. Compare them when application behavior disagrees with dig.
Test both A and AAAA. Broken IPv6 can be isolated with:
curl -4 -v https://api.example.com/health
curl -6 -v https://api.example.com/health
Do not directly edit generated /etc/resolv.conf; fix Netplan or the active network manager.
6. Is the service listening on the right address?
sudo ss -lntup
sudo ss -lntp 'sport = :443'
sudo ss -lnup 'sport = :53'
127.0.0.1:8080is reachable only from that network namespace.0.0.0.0:8080listens on all IPv4 addresses.[::]:8080listens on IPv6; dual-stack behavior depends on configuration.
systemctl status nginx --no-pager
journalctl -u nginx --since '-15 min' --no-pager
An active service may still bind the wrong address, have a failed upstream, or lose workers.
7. Understand refused, timeout, and reset
- Connection refused: the host returned TCP RST, commonly due to no listener or firewall reject.
- Timeout: no response, possibly due to routing, packet drop, security group, NAT, or a down host.
- Connection reset: a peer forcibly closed an established connection; inspect application, proxy, and TLS logs.
- No route to host: no route or an unreachable response, sometimes generated by a firewall reject.
nc -vz -w 3 api.example.com 443
curl -v --connect-timeout 3 https://api.example.com/health
curl -v separates DNS, selected IP, connect, TLS, and HTTP phases. Do not use -k as a TLS fix; it only hides certificate validation failures.
8. Check every firewall layer
sudo ufw status verbose
sudo nft list ruleset
sudo iptables-save
Beyond the host, inspect cloud security groups, network ACLs, load balancers, physical firewalls, and Kubernetes policies. Never flush a production firewall over remote SSH. Make narrow changes and retain an out-of-band path.
sudo ufw logging medium
journalctl -k --since '-10 min' | grep -i 'UFW'
Log selectively; logging every packet can create serious I/O.
9. Test HTTP and TLS with the correct hostname
curl -v https://api.example.com/health
curl --resolve api.example.com:443:203.0.113.20 \
https://api.example.com/health
openssl s_client \
-connect 203.0.113.20:443 \
-servername api.example.com \
-showcerts </dev/null
--resolve forces an IP while preserving SNI and Host, making it useful for testing an origin before DNS cutover. Calling https://IP can select the wrong certificate and virtual host.
Inspect chain, SAN, expiry, SNI, and redirects. A 502/504 means client-to-proxy networking may be healthy; test proxy-to-upstream from the proxy's own namespace.
10. Measure path and packet loss carefully
tracepath api.example.com
traceroute -T -p 443 api.example.com
mtr -rwzc 50 api.example.com
Intermediate routers may suppress ICMP while forwarding normally. An asterisk alone is not failure. Intermediate packet loss matters when it persists through later hops and the destination. Test the real TCP port and compare from multiple source networks.
11. MTU and Path MTU Discovery
An MTU mismatch can allow small pings while large HTTPS requests or uploads stall, especially across VPNs and overlays.
ip link show dev eth0
tracepath api.example.com
ping -M do -s 1472 -c 3 api.example.com
For IPv4, 1472 plus 28 bytes of headers corresponds to MTU 1500. IPv6 and tunnels differ. Reduce the payload to find the threshold, and verify the complete path before changing MTU. Blocking ICMP “fragmentation needed” can break PMTUD.
12. Use tcpdump to locate the packet
sudo tcpdump -ni any host 203.0.113.20 and port 443
sudo tcpdump -ni eth0 'tcp port 443 and (tcp[tcpflags] & tcp-syn != 0)'
sudo tcpdump -ni any port 53
- SYN leaves with no SYN-ACK: path, drop, or nonresponsive server.
- SYN followed by RST: rejected port or no listener.
- No outgoing SYN: resolver, route, local firewall, or application issue.
- Server sends SYN-ACK but client never sees it: return path, asymmetry, or upstream firewall.
sudo timeout 30 tcpdump -ni any -s 0 \
-w /tmp/network-issue.pcap \
'host 203.0.113.20 and port 443'
Packet captures can contain credentials, cookies, and personal data. Limit filters and duration, protect the file, and delete it securely afterward.
13. Containers and network namespaces
Host tests are insufficient when the process runs in another namespace:
docker exec -it app sh
ip address
ip route
cat /etc/resolv.conf
getent hosts database
nc -vz database 5432
Inside Docker, 127.0.0.1 refers to the container itself. Use Compose service names. Inspect membership, published ports, and namespaces:
docker network inspect app_default
docker port app
sudo nsenter -t PID -n ss -lntp
For Kubernetes, inspect pod DNS, Service selectors/endpoints, NetworkPolicy, and test from a debug pod in the same namespace.
14. Connection tracking and ephemeral ports
ss -s
ss -ant state time-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range
sudo conntrack -S
dmesg | grep -i conntrack
High traffic can exhaust conntrack entries, ports, or sockets. Do not copy random TIME_WAIT/sysctl tweaks. First find connection churn, missing keep-alive, NAT bottlenecks, or incorrect pools; then capacity-plan limits.
15. Change remote network configuration safely
sudo netplan generate
sudo netplan try
netplan try requests confirmation and can roll back on lost connectivity. Critical servers still need console or out-of-band access. Back up configuration, validate YAML, and make one small change at a time.
Ten-minute playbook
ip -br linkandip -br addr: link and address.ip route get DEST: selected route, source, and interface.ip neighand gateway reachability.digplusgetent: DNS versus system resolver.ss -lntup: listener address and port.nc/curl -v: refused, timeout, TLS, or HTTP.- Host, cloud, and container firewalls.
curl --resolve: isolate DNS from HTTPS/SNI.tcpdumpon client and server: locate the stopping point.- Change only after forming a hypothesis and rollback plan.
Conclusion
Effective Linux network troubleshooting is ordered elimination: link, IP, route, neighbor, DNS, socket, firewall, transport, TLS, then application. Each tool answers a different question; no single command proves that “the network is fine.” Testing from the correct namespace, observing both directions, and preserving evidence before changes turns apparently random incidents into a specific broken point.




No comments yet. Be the first to share your thoughts.