IPv6 Troubleshooting

Joni Julian

Ohio LinuxFest

September 14, 2013

I Want You To Use IPv6

IPv6 Poster

IPv6 infographic

IPv6 Basics

128-bit addresses as 8 hexadecimal fields separated with colons, not those mere 32-bit addresses of IPv4
no broadcast, just different kinds of multicast (L2 and link-local work well) and unicast
no private addresses, just link-local with low TTL and now ULA / unique local addresses from RFC 4193 that can't be routed globally (no NAT loophole either)
originally, no DHCP, just automatic configuration but now DHCPv6 too
- what does DHCP do well? it configures additional parameters like DNS and WINS, so DHCPv6 is good for enterprises
IP Security built-in

IPv6 Packet Header

from Wikipedia

IPv6 Packet Header

flow labelling for QoS or netflow v9+
payload length, next header (for options), hop limit (TTL) are needed
fewer fields, everything else is optional (no more rarely-used options from the IPv4 packet header)

IPv6 Scopes

loopback: local NIC ::1
link local: (V)LAN only, automatic, unrouted fe80::/10
site local: fec0::/10, deprecated for ULA using fdxx::/8
global unicast: IANA-assigned unique address
- documentation: 2001:db8::/32
anycast: one-to-nearest, routers only

IPv6 Scopes for Multicast

multicast: ff00:/8 and again with the scopes ...
- ff01::1: interface local, all nodes
- ff02::1: link local, all nodes
- ff01::2: interface local, all routers
- ff02::2: link local, all routers
- "There's a multicast group for that!"

IPv6 Addresses

A wise co-worker once told me that if you have a handle on IPv6 addresses, you're most of the way to understanding IPv6.

I have two favorite tools to decode IPv6 addresses, tavian and v6decode.
- You can also install ipv6calc on *nix
Down one layer, the IPv6 ethertype is 0x86DD

IPv6 Addresses

You should be able to recognize some IPv6 address types on sight.
- ::1 loopback
- fe80::xxxx/10 link-local unicast (VLAN)
- x:OUIff:feSN EUI-64 interpretation of host's MAC address
- 2002::/16 the accepted 6to4 range from IANA but some older versions appear to use 2000 or 2001 if I read SubnetOnline right: just ask tavian or v6decode; 6to4 relay uses 192.88.99.1, the 6to4 anycast IPv4
- ff00::/8 multicast

IPv6 Support

University of Wisconsin at Madison list of operating systems supporting IPv6

IPv6 Performance

native IPv6 performance is best

IPv6 Performance

Since "prefer v6" goes 6 then 4, there may be a timeout. Web pages may be partly or entirely blank, and IE has it the worst since it doesn't render as pieces arrive.

What's My Address?

Unix: still ifconfig of course
Linux: also ip address show
Windows: ipconfig or try netsh interface ipv6 show address

hope@calvin$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:14:4F:01:A6:DA 
 inet addr:132.91.236.15 Bcast:132.91.239.255 Mask:255.255.252.0
 inet6 addr: 2001:db8:1000:2001::4/64 Scope:Global
 inet6 addr: fe80::214:4fff:fe01:a6da/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:7033253 errors:0 dropped:0 overruns:0 frame:0
 TX packets:3272993 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000 
 RX bytes:776277559 (740.3 MiB) TX bytes:718801387 (685.5 MiB)

Many Other IPv6 Addresses

IPv6 can have random addresses that change often thanks to RFC 4941 - Privacy Extensions for Stateless Address Autoconfiguration in IPv6 (the update to RFC 3041)

this FAQ entry to disable it in Windows
Linux IPv6 Support to enable it in Linux

Some Corollaries

IPv4	IPv6
ARP	NDP
IGMP	MLD

Troubleshooting with OSI Layers

	IPv4	IPv6
1	no change
2	`arp`	`ndp` is part of ICMPv6
3	`ping`, `traceroute`, `host`	`ping6`, `traceroute6`, `host -6`
4	no change, `netstat`
7	`iptables`	`ip6tables`

ICMPv6

You can't turn it off!

No Broadcast

IPv6 only uses multicast!

Ethernet multicast to 33:33 + last 4 octets of destination IPv6 address (ref) instead of ff:ff:ff:ff:ff:ff for Ethernet broadcast
the solicited node multicast address is ff02::1:ffxx + last 3 octets of destination IPv6 address + /104
IPv6 link-local (VLAN) all hosts multicast address is ff02::1 instead of 255.255.255.255 broadcast

NDP

NDP, Neighbor Discovery Protocol, is part of ICMPv6 from RFC 4443
NDP starts with a neighbor solicitation (request, ICMPv6 info 135), answered by neighbor advertisement (reply, ICMPv6 info 136)

NDP

sent to solicited-node multicast address of ff02::1:ff + last 24 bits/3 octets of IPv6 unicast address + /104
- calvin has 2001:db8:1000:2001::4 or 2001:0db8:1000:2001:0000:0000:0000:0004 with zeroes, so its solicited node multicast address is ff02::1:ff for multicast plus 00:0004 for calvin so ff02::1:ff00:0004 or ff02::1:ff00:4 without leading zeroes
includes NUD, Neighbor Unreachability Detection

ndp and ndisc6

ndp hostname

 -a (all)
 -d (delete)
 -n (numeric, skip name resolution)

hope@calvin$ sudo ndisc6 hobbes6 eth0
Soliciting hobbes6 (2001:db8:1000:2001::5) on eth0...
Target link-layer address: 00:14:4F:01:29:8C
 from 2001:db8:1000:2001::5

ndp alternative

If ndp isn't present, use ip -f inet6 neighbor or ip -6 neighbor show (may use neigh instead)

hope@calvin$ ip -f inet6 neigh | grep -v router
fe80::214:4fff:fe01:298c dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
2001:db8:1000:2001::5 dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
hope@calvin$ ip -f inet6 neigh
2001:db8:1000:2001::2 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
fe80::214:4fff:fe01:298c dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE
fe80::201:d7ff:fe68:fec4 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
2001:db8:1000:2001::1 dev eth0 lladdr 00:01:d7:68:fe:c4 router REACHABLE
2001:db8:1000:2001::3 dev eth0 lladdr 00:01:d7:73:66:c4 router REACHABLE
fe80::201:d7ff:fe73:66c4 dev eth0 lladdr 00:01:d7:73:66:c4 router REACHABLE
2001:db8:1000:2001::5 dev eth0 lladdr 00:14:4f:01:29:8c REACHABLE

prefer v6

When we added IPv6 to calvin's DNS registration, only non-Windows users could ssh in. So I would describe Microsoft's version of "prefer v6" to be "fail badly if IPv6 isn't working" when Linux and Mac failed over to IPv4 silently and quickly under the same circumstances. (My guess is that Mac and Linux listened to the NUD message, and Windows is just a nudnik.)

ping6, traceroute6

I added IPv6 to DNS on calvin. When I went to test it, I discovered I could ping6 and host -6 calvin from hobbes, but not from susie. The first two have static IPv6 addresses and are in the same F5 VLAN, while susie has an automatic IPv6 address (SLAAC) for its VLAN. I also know susie's IPv6 can reach the outside IPv6 world. I suspected the VLAN, so I turned to our Layer 3 tools.

hope@calvin$ host -6 -t AAAA www.mysite.edu calvin6
Using domain server:
Name: calvin6
Address: 2001:db8:1000:2001::4#53
Aliases: 

www.mysite.edu has IPv6 address 2001:db8:1000:2001::11

on calvin6

hope@calvin$ host calvin
calvin.net.mysite.edu has address 132.91.236.15
calvin.net.mysite.edu has IPv6 address 2001:db8:1000:2001::4
hope@calvin$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:14:4F:01:A6:DA
 inet addr:132.91.236.15 Bcast:132.91.239.255 Mask:255.255.252.0
 inet6 addr: 2001:db8:1000:2001::4/64 Scope:Global
 inet6 addr: fe80::214:4fff:fe01:a6da/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:53237 errors:0 dropped:0 overruns:0 frame:0
 TX packets:46730 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:4034756 (3.8 MiB) TX bytes:8427890 (8.0 MiB)

on calvin6

hope@calvin$ ping6 -c 4 2001:db8:1000:2001::1
PING 2001:db8:1000:2001::1(2001:db8:1000:2001::1) 56 data bytes
64 bytes from 2001:db8:1000:2001::1: icmp_seq=0 ttl=64 time=0.150 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=1 ttl=64 time=0.205 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=2 ttl=64 time=0.165 ms
64 bytes from 2001:db8:1000:2001::1: icmp_seq=3 ttl=64 time=0.171 ms

--- 2001:db8:1000:2001::1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.150/0.172/0.205/0.025 ms, pipe 2
hope@calvin$ ping6 -c 4 hobbes
PING hobbes(2001:db8:1000:2001::5) 56 data bytes
64 bytes from 2001:db8:1000:2001::5: icmp_seq=0 ttl=64 time=1.43 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=1 ttl=64 time=0.120 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=2 ttl=64 time=0.128 ms
64 bytes from 2001:db8:1000:2001::5: icmp_seq=3 ttl=64 time=0.135 ms

--- hobbes ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.120/0.453/1.432/0.565 ms, pipe 2

hobbes6 to calvin6

hope@hobbes$ ping6 -c 4 calvin
PING calvin(2001:db8:1000:2001::4) 56 data bytes
64 bytes from 2001:db8:1000:2001::4: icmp_seq=0 ttl=64 time=0.125 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=1 ttl=64 time=0.134 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=2 ttl=64 time=0.104 ms
64 bytes from 2001:db8:1000:2001::4: icmp_seq=3 ttl=64 time=0.114 ms

--- calvin ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.104/0.119/0.134/0.013 ms, pipe 2

susie to calvin6

hope@susie$ host susie
susie.net.mysite.edu has address 132.90.145.183
hope@susie$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr E4:1F:13:2C:80:18
 inet addr:132.90.145.183 Bcast:132.90.255.255 Mask:255.255.255.0
 inet6 addr: 2001:db8:1000:5001:e61f:13ff:fe2c:8018/64 Scope:Global
 inet6 addr: fe80::e61f:13ff:fe2c:8018/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:676169 errors:0 dropped:0 overruns:0 frame:0
 TX packets:84130 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:63870876 (60.9 MiB) TX bytes:10642501 (10.1 MiB)
 Interrupt:169 Memory:92000000-92012800

susie to calvin6

hope@susie$ ping6 -c 4 2001:db8:1000:2001::4
PING 2001:db8:1000:2001::4(2001:db8:1000:2001::4) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- 2001:db8:1000:2001::4 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms
hope@susie$ ping6 -c 4 calvin
PING calvin(2001:db8:1000:2001::4) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- calvin ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3001ms

susie to calvin6's gateway

hope@susie$ ping6 -c 4 2001:db8:1000:2001::1
PING 2001:db8:1000:2001::1(2001:db8:1000:2001::1) 56 data bytes 
From 2001:db8:1000:24::1 icmp_seq=0 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=1 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=2 Destination unreachable: Address unreachable 
From 2001:db8:1000:24::1 icmp_seq=3 Destination unreachable: Address unreachable

--- 2001:db8:1000:2001::1 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3001ms

calvin6 to susie

hope@calvin$ ping6 -c 4 2001:db8:1000:5001:e61f:13ff:fe2c:8018
PING
2001:db8:1000:5001:e61f:13ff:fe2c:8018(2001:db8:1000:5001:e61f:13ff:fe2c:8018)
56 data bytes

--- 2001:db8:1000:5001:e61f:13ff:fe2c:8018 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

traceroute6 between the two VLANs

hope@susie$ traceroute6 2001:db8:1000:2001::5
traceroute to 2001:db8:1000:2001::5 (2001:db8:1000:2001::5), 30 hops max, 40 byte packets
 1 (2001:db8:1000:5001::1) 0.594 ms 0.616 ms 0.713 ms
 2 (2001:db8:1000:24::1) 0.750 ms 0.837 ms 0.912 ms
 3 (2001:db8:1000:24::1) 0.997 ms !H 1.096 ms !H 1.166 ms !H
hope@susie$

hope@hobbes$ traceroute6 2001:db8:1000:5001:e61f:13ff:fe2c:8018
traceroute to 2001:db8:1000:5001:e61f:13ff:fe2c:8018 (2001:db8:1000:5001:e61f:13ff:fe2c:8018),
 30 hops max, 40 byte packets
 1 (2001:db8:1000:5001:e61f:13ff:fe2c:8018) 0.211 ms 0.202 ms 0.171 ms
 2 (2001:db8:1000:5001:e61f:13ff:fe2c:8018) 0.723 ms 0.621 ms 0.821 ms
 3 * * *
 . . .
30 * * *
hope@hobbes$

netstat

hope@calvin$ netstat -an | grep "Proto\|2001"
Proto Recv-Q Send-Q Local Address Foreign Address State 
tcp 0 0 2001:db8:1000:2001::4:2220 2001:db8:1000:5001:221:58302 ESTABLISHED 
tcp 0 0 2001:db8:1000:2001::4:2220 2001:db8:1000:5001:221:58502 ESTABLISHED 
udp 0 0 2001:db8:1000:2001::4:53 :::*  
udp 0 0 2001:db8:1000:2001::4:123 :::*

TCPv6

Most higher-level protocols, like TCP and UDP, are unchanged by IPv6, but some, like FTP and NTP, need specific updates. (ref) The checksum calculation does change for TCP and UDP, though.

ip6tables

hope@calvin$ sudo service ip6tables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination 
1 RH-Firewall-1-INPUT all ::/0 ::/0 

Chain FORWARD (policy ACCEPT)
num target prot opt source destination 
1 RH-Firewall-1-INPUT all ::/0 ::/0 

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination 

Chain RH-Firewall-1-INPUT (2 references)
num target prot opt source destination 
1 ACCEPT all ::/0 ::/0 
2 ACCEPT icmpv6 ::/0 ::/0 
3 ACCEPT esp ::/0 ::/0 
4 ACCEPT ah ::/0 ::/0

ip6tables

5 ACCEPT udp ::/0 ff02::fb/128 udp dpt:5353 
6 ACCEPT udp ::/0 ::/0 udp dpt:631 
7 ACCEPT tcp ::/0 ::/0 tcp dpt:631 
8 ACCEPT udp ::/0 ::/0 udp dpts:32768:61000 
9 ACCEPT tcp ::/0 ::/0 tcp dpts:32768:61000 flags:!0x16/0x02 
10 ACCEPT tcp ::/0 ::/0 tcp dpt:8080 
11 ACCEPT tcp ::/0 ::/0 tcp dpt:22 
12 REJECT all ::/0 ::/0 reject-with icmp6-adm-prohibited

hope@calvin$ sudo service ip6tables status
Firewall is stopped.

Cookbook

ARIN's wiki page on troubleshooting IPv6 issues

Summary

v6decode IPv6 address tool
tavian IPv6 address tool
UWM list of O/S support
UWM list of network troubleshooting tools like ping and ping6
random addresses (privacy), Windows off or Linux on ... personally I like the EUI-64 embedding the MAC address in automatic (SLAAC) IPv6 addresses!

Conclusion

All of your network troubleshooting instincts are still valid for IPv6, but some of the commands are updated.

Don't fear the

More Information

Flying Penguin Technologies to teach IPv6 (with blog, and this talk possibly here and earlier version here)
ARIN and their wiki page on troubleshooting IPv6 issues
RIPE has a lot of good information