7:55 is very IMPORTANT reminder. MTU is per hop, while MSS, since it is on TCP, it's end-to-end. Networking people (me too) sometimes jump the gun that when they pinged next hop and able to get 1500 bytes MTU (do not fragments), immediately assume it is the same treatment along the hops throughout the network. I remember dealing with MPLS backbone team and they assumed basically that same thing. UNTIL we went P routers by P routers pinged each hop and identified there was an issue where somehow the MTU on that hop was dropped down to ~400 bytes. Once the "expert" backbone team rerouted the traffic away from that backbone path, then the application can resume.
It was really an eye opener. I have read a book on ccnp by Bradley another author I respect so much. He actually recommended tunnels to be set ip mtu to 1400 and mss 1360. But it was until today .I know why .also we have other encapsulation like mpls ,gre that has their sizes that need to be put into consideration. Thanks Bombia and Chris .
Yes, more content with Chris and Wireshark. The real world examples are great and really helps show the thought process for going through the capture. It is also great how you go back and forth and ask questions that we the viewer may ask.
// MENU // 00:00 ▶ Coming Up 00:25 ▶ Intro 00:32 ▶ Chris introduction 00:47 ▶ Topic: Maximum Segment Size (MSS) 01:27 ▶ Explaining Maximum Transmission Unit (MTU) 08:42 ▶ Interface layout 10:25 ▶ David Bombal "War Story" 12:00 ▶ Wireshark demo 13:26 ▶ Increasing the MTU on your device for larger connections 16:27 ▶ Difference between MTU and MSS 19:36 ▶ Wireshark demo (cont'd) 24:58 ▶ Using Path MTU Discovery 27:02 ▶ Ping and Wireshark demo 33:32 ▶ Cool trick for Mac system 35:08 ▶ TCP/MSS Clamping 38:21 ▶ Chris Greer "War Story" 51:09 ▶ What happens if you can't capture a server 55:08 ▶ MSS Adjustment commands 56:55 ▶ Tunnel Path MTU Discovery 57:40 ▶ Figuring out 1432 01:02:52 ▶ Conclusion 01:04:48 ▶ "Cool features" in Wireshark Previous video: ua-cam.com/video/rmFX1V49K8U/v-deo.html // Wireshark PCAP files // MTU PCAP: github.com/packetpioneer/youtube/blob/main/PMTUD.pcapng War Story PCAP Client: github.com/packetpioneer/youtube/blob/main/slowfile-clientside.pcapng War Story PCAP Server: github.com/packetpioneer/youtube/blob/main/slowfile-serverside.pcapng Special “Thumbs Up” and “Subscribe” PCAP: github.com/packetpioneer/youtube/blob/main/tcp-streamexport.pcapng // GOOD READING // Network Implications of PMTUD: www.ipspace.net/kb/Internet/PMTUD/30-network-implications.html Path MTU Discovery: www.ipspace.net/kb/Internet/PMTUD/20-mtu-discovery.html Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPsec: www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html Configuring TCP MSS Adjustment: www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/16-9/configuration_guide/ip/b_169_ip_3650_cg/configuring_tcp_mss_adjustment.pdf Ethernet MTU and TCP MSS Adjustment Concept for PPPoE Connections: www.cisco.com/c/en/us/support/docs/ip/transmission-control-protocol-tcp/200932-Ethernet-MTU-and-TCP-MSS-Adjustment-Conc.html // MY STUFF // www.amazon.com/shop/davidbombal // David SOCIAL // Discord: discord.com/invite/usKSyzb Twitter: twitter.com/davidbombal Instagram: instagram.com/davidbombal LinkedIn: www.linkedin.com/in/davidbombal Facebook: facebook.com/davidbombal.co TikTok: tiktok.com/@davidbombal UA-cam: ua-cam.com/users/davidbombal // Chris SOCIAL // Udemy course: davidbombal.wiki/chriswireshark LinkedIn: www.linkedin.com/in/cgreer/ UA-cam: ua-cam.com/users/ChrisGreer Twitter: twitter.com/packetpioneer // VLAD SOCIAL // Twitter: twitter.com/Packet_vlad PMTUD Blog: www.packettrain.net/2016/09/21/pmtud-or-not/ Thanks Vladimir Gerasimov! // SPONSORS // Interested in sponsoring my videos? Reach out to my team here: sponsors@davidbombal.com Please note that links listed may be affiliate links and provide me with a small percentage/kickback should you use them to purchase any of the items listed or recommended. Thank you for supporting me and this channel!
i just want to say thank you from the bottom of my heart lots of love david ... i am getting started in cyber security and i have everything i wanted here in your channe thanks for your effort ......
Great video! I had a client running a Cisco 1100 in their home office - everything was working perfectly except for Netflix and Disney+ which seemed very strange. After a fair bit of investigation the issue turned out to be the MSS. They were on a home internet plan using pppoe so had to use IP TCP ADJUST-MSS 1452 command to account for the 8 byte overhead of pppoe.
I like that you explained the battle and the winning command so if somebody has this problem in the field they can also use your strategy to fix it. God bless you
Amazing job, guys ! We see a lot of information written on web but when we see a live demo it's much more interesting and captivating. I've recently found David and Chris's channels and can't stop watching. Keep doing wireshark analyses for network problems ! It's awesome !
This was a great video. It reminded me and issue that I was dealing with around 2 years ago with Aruba APs. Our client has setup like this. One local WLAN aruba controller on site Spain and one redundant controller which was on remote site in Sweden. Clients reported issue with slow wifi. I found out that clients that are connected to APs which are connected to remote controller experience slowness. Other users connected to APs which used local wlan controller had no issue. So we started live troubleshooting and see that ping was ok but there was real slowness when opening web page or doing skype call (user was green but with packet loss). We started to manipulate ping request with more bytes and then packet loss appeared. Later I found out that APs has set "do not fragment" bit by default and we did calculation with wan team that packet is too big. So on our side we turn on preemption which will move APs to local WLC if it is online and WAN team did their config to divide packets to smaller size. This was really great issue and I enjoyed it :) Btw. redundat remote wlc served for several small sites. Usually medium and big sites had 2 local WLCs but because of fact that some customer sites were too small they had setup with one local and one remote wlc.
Wow. I'm indeed wowed. This is an impressive and exceptional delivery. It's unbelievable to find this kind of quality content for free. I felt I was in a classroom. I would appreciate a deep dive on the entire TCP/IP and Ethernet header. Thanks for putting this out.
David and Chris, you won't believe how many networking professionals have no understanding on MTU or MSS, so thank you so much for this! Please more on TCP and ideally the next with some focus on TTL (You touched on it in this video "My TTL started at 64 and it's now 54, so I'm going through 10 Routers") Great video, subscribed and liked and will check out Chris' UA-cam channel
Anohter fantastic collaboration and learning time. And the 'War Story" shows a real-world example for us to dive into and understand. Thank you David and Chris!
Thank you so much Chris and David. This video is a goldmine and I learnt a lot from it. Watching it over and over again, and it just keeps making more sense.
I love the way you both engaged us all. I have really struggled in and out to know mtu and mss in detail. Now, I’m all good and proud. Thanks for the video. Keep up the great work!
Great video guys, very explanatory and easy to comprehend, I confirm this TCP issues are nigthmares, having dealing with a similar problem months ago where one of the branch offices experience a low vpn throughput and fragmentation, the issue was caused by incorrect mss clamping value, it took few days to figure out whats was causing it. Like Chris said, its easier to find out what's wrong when the path is broken rather that in this trickier issues. Thanks for sharing your vastly knowledge
These videos answer the question: what if Robin Williams was a network engineering vlogger? In all seriousness though, great content; appreciate the clear and real-world explanations!
Excellent content, Thanks again for sharing this valuable knowledge specially for newbies like me. I've been working on learning and mastering the fundamentals and this material makes it possible. Thanks Again.
Hi David & Chris, Thank you for sharing your experience and knowledge. Now I understand why MTU & MSS-adjust commands are used in the mGRE tunnels comes into play. Thanks alot guys and i can't wait to see you twos in your next collaboration. Cheers! 🇵🇬
Thanks, David. This hour lesson had so much information that I had to watch in slow-mode and analyse it for about 7 days! true gold. Thanks again and I am subscribing to Chris' channel as well. Have a great day.
David combines real war story to explain the symptoms we will experience if we have MTU and MSS issue. that is cool. Connects the theory with real life. More war stories please, that makes technical video more interesting.
Awesome guys, you guys made my day..First time watching a long video of yours, I hope this will help a lot. Thanks, Chris and David for teaching new kinds of stuff in TS.
If anyone is confused when Chris is explaining about ping capping at 1472 bytes because you have the IP and ICMP headers on top, but wondering what happened to the TCP header that he was talking about earlier in the video and why that isn't subtracted, remember that ICMP is a completely different L4 protocol. Hence in this example we do not need to take into account the 20 byte TCP header, just the ICMP and IP headers. Layer 4 - UDP, TCP, ICMP
Thank you for the review. I ve seen this in the 2000 during ATM implementations and many tunnel and datalinks. it is a great video to review and or learn in a practical matter what you just setup as default as MTU 1500 in your routers. and as you described to know why datalinks are slow when pings show Ips are reachable
Wow, David! What a blast from the past! That Cisco MTU/MSS white paper you showed a clip of was my "go to" doc for many years for helping to explain fragmentation issues to customers and peers alike. I still have the link to it in my bookmarks, since you never know when MTU's are gonna raise their ugly heads! As a long-time CCIE (#1937 just over 26 years ago) your content is refreshingly good!
Man this is gold, I'd love to see that video about 10Gpbs and 100Gbps connections and MTU tweaking, I happen to work at a service provider and there are so many customers out there that complain about running speed test on 10Gbps links while using laptops and not getting to the right speeds, of course at times there are more issues in the middle that can produce such outcome but most of the time is just either speed and duplex or MTU on the PE side that may cause things of that nature, at least in my experience.
Having same problem as you, but commonly laptops are coming capped to 1Gbps NIC card, so even incrementing their MTU won't benefit as they won't be able to test above 1gbps.
Thank you both for this detailed video. Appreciate the time and effort both put into for this interview! At many places I was lost - if the header bytes should be added or not. Should watch this video at-least twice to comprehend the depth of knowledge that this video contains. Pls continue your interviews with Chris. Absolutely love them and learning a lot from them. Looking forward for more Wireshark deep dive videos.
Really fantastic content. Thank you so much! I had network technology and Cisco trainings in my education two times, but it was always boring theory an I thought, it's a hard way to go any deeper in this stuff. But your videos show me, that it's all about good coaches.
One point about the Don't Fragment flag. Windows only sets it on TCP, but Linux does it for everything, IIRC. This can lead to a situation with Windows where TCP works fine, but there may be problems with UDP. On the other hand, Linux will not experience the UDP problem. Also, Path MTU Discovery is mandatory with IPv6, as fragmentation is not allowed.
Excellent video, been running into this a lot lately in a number of scenarios when replacing branch routers, first thing that breaks is Eap-TLS for wireless, those certificates do not like fragmentation, so have had to adjust MTU in a bunch of places.
this is teaching 2.0. You can read school books, learn all the commands and theories as much as you can but this is both fun ánd interesting AND you even learning more then ever. Thanks guys
Had this issue with MPLS WAN circuits. We ran GRE over the MPLS circuits to carry our own EIGRP routing traffic. We had to do MSS "clamping" to account for the extra overhead from our GRE tunnel. We added another MPLS provider circuit to our headend and had to do the same but with a lower value to those locations using that MPLS provider. All to do with them using space in the TCP header for VLAN/MPLS.
12:44 Important note. For perfection, system can receive maximum. The reason for that is to avoid fragmentation. As we know, nowadays lots of applications use DF (do not fragment) bit. So, this is the reason to use standard number 1460 bytes.
Cool instead of just disabling ipv6 on a router I now know what the reason could be when pages aren't loading on those random clients that uses corporate VPNs. MTU settings, never knew thanks :) Valuable information as always!
I already have experienced this in a large prod. environmen network where too much packet drops were coming and finally we were able to figure out the root cause to be the MTU size. Gr8 video.
@@davidbombal It happened on a Fortigate firewall, and after 2 days of troubleshooting and banging our heads and involving their tech support, this was identified. But still, no one was still able to identify where the MTU was getting changed in the packets complete path. I hope this video will be a good starting point for many. Kudos, and please keep up the good work!
Great video guys. Something is throwing me off, around the 33:22 mark when you're doing a 1472 df ping and it's replying at 1480, but Wireshark is capturing only 1464 for data + 8 for ICMP. Aren't we missing 8 bytes somewhere?
I noticed this discrepancy too, so I did some research to identify the answer: There is an 8-byte 'Timestamp' field shown by Wireshark right above the 'Data' payload section. This 'Timestamp' field is technically part of the ICMP payload, but is distinguished from the random data portion of the ICMP payload that is shown inside the 'Data' payload section. You can see this for yourself by reproducing the ping + capture test demonstrated in the video; as you select each field of the ICMP header within the 'Packet Details' panel, pay attention to the highlighted characters within the 'Packet Bytes' panel - each individual character in this panel represents a single byte. Alternatively, the status bar located at the bottom left of the Wireshark application window contains a field description that also includes the byte size of the field. By performing this exercise you can manually add up 8 total bytes of header + 8 total bytes of the Timestamp payload data + the 1464 bytes of random payload data, which equals 1480. And then of course you just add the 20 bytes from the IP header to get the total Ethernet MTU size of 1500 bytes.
I have a pretty basic doubt on Chris' war story, where one of the routers on the client side did not have mss clamping enabled. In such a setup, shouldnt the cisco router on the server side, which had mss clamping enabled, clamped the MSS on the SYN packet from the client as well? I understand that when MSS clamping is enabled on an interface, it applies for both outbound and inbound traffic. Is it not the case?
very interesting content: generally, to avoid fragmentation on some routers it's possible to use MSS clamping. E.g., if the router has more than two interfaces, on Cisco IOS this is possible using bidirectionally "ip tcp adjust-mss" interface configuration command. :-)
Thanks for a good video on this interesting topic. A quick comment though, at about 14:00 you seem to imply that you can/should change the local NIC MTU to some arbitrary higher value. However, that cannot work if not all switch ports and the router interface on the same VLAN also have a non-default higher MTU. Additionally, if you will not have the full path, end-to-end, with higher MTU then changing the local LAN will likely not improve much.
Thank you for this video! Could you please explain the difference between MTU and the calculated window size? They both seem related to the largest size a machine can send but I don't understand the contrast.
Thanks (again) for this very interesting video. Something is still unclear to me however : on the Chris explanation at @1:00:00, he's saying that the root-cause was because they configured clamping while it was not necessary (if we can send 1472 bytes ping, we should be able to send 1460 bytes TCP segments). However, in this case, why were those "large" segments dropped ? MSS Adjust just changes the MSS value on the TCP SYN packets, but it does not drops further packets received being "above" this MSS value, am I wrong ?
Think he went off on a tangent, but wasn't allowed to test further to check his theory, but agree there's something in between dropping the packets. I'm guessing the original clamp was put in for a reason, but perhaps the client side was a new or replacement router, and the MTU issue was forgotten about.
Probably the answer is because ICMP (8B header) runs on top of IP (20B header), so when you ping and set the packet size (ping -s 1472) you set only the ICMP payload not including ICMP header (IP[20B] + ICMP[8B] + ICMPpayload[1472] = 1500B). Just remembering that ICMP with do not fragment tests MTU end-to-end and doesn't care about MSS as MSS is TCP only. Then a ping with 1472 would do perfectly fine. TCP is another thing as it adds 20B header on top of IP, so the math would be IP(20B) + TCP(20B). A safety rule of thumb for me would be to subtract from biggest successful ping payload size extra 12B for accomodating TCP. Example: if a ping -s 1472 to a destination was successful, the MSS adjustment would be 1472 - 12 (TCPheader-ICMPheader) = 1460. Just remembering here that IP header and TCP header sizes could vary (20B to 60B). For the worst case calculation (IP[60B] + TCP[60B]) the math would be 1472B (biggest successful ping payload) - 40B IP(adding extra 40B to already existing 20B) - 52 TCP(removing the 8 bytes from ICMP header that were already considered in ping) = 1380B Not considering in this calculation any extra encapsulation overhead as VXLAN, etc...
Hi David after i finish up my video about the pure basics for computers in my mission to join the IT teaching community i will watch the full intervieuw :)
Great Video David! Pretty awesome series for understanding TCP. Just one question though, are you planning on releasing more videos in the series? These videos are a great help and would love to know if we have more coming. Anyways, thanks for all the content. Cheers :)
I ran into these issues while using Aruba 8 Access Points traveling over Meraki VPN back to a controller via GRE. Had to reduce the AP MTU down to 1360 to stop the fragmentation.
Hi Gents…I need some help in further clarifying this ping with 1472 df. 1) My understanding is that the specified size in the ping command is the actual payload (icmp header + data = 8+1464 = 1472),but without IP header.But where are then those another 8 bytes not accounted for? (basically I see it as 1492,not 1500 ) 2) When pinging from a router instead,,the size is actually including the IP Header.For example,a ping size of 1500 will give you a Icmp Data (1472) + Icmp H (8) + IP (20) = 1500 ,with 1514 length captured (plus L2) So then…is this behavior regarding the ping size different between host/servers and routers?
Great video, but one thing doesn't make sense in the WAR story. Just because MSS was adjusted on one side wouldn't necessarily cause an issue, correct? Wouldn't that right hand router also have to have lowered it's MTU?
So with a slow network situation as the example in the video.... Would you be able to test if a MTU change would work by setting the local device MTU in the network settings? I'm just thinking that the initial SYN packet should then contain a lower value so the server would try smaller packets immediately. Or am I missing something here.
Firewalls that NAT also contribute if the bottleneck is after the firewall. The ICMP packet that comes back comes from a different IP than the destination and therefore gets dropped by the firewall ruleset. And because of the NAT, there is no way to figure out the connection it should go back to.
7:55 is very IMPORTANT reminder.
MTU is per hop, while MSS, since it is on TCP, it's end-to-end.
Networking people (me too) sometimes jump the gun that when they pinged next hop and able to get 1500 bytes MTU (do not fragments), immediately assume it is the same treatment along the hops throughout the network.
I remember dealing with MPLS backbone team and they assumed basically that same thing.
UNTIL we went P routers by P routers pinged each hop and identified there was an issue where somehow the MTU on that hop was dropped down to ~400 bytes.
Once the "expert" backbone team rerouted the traffic away from that backbone path, then the application can resume.
It was really an eye opener. I have read a book on ccnp by Bradley another author I respect so much. He actually recommended tunnels to be set ip mtu to 1400 and mss 1360. But it was until today .I know why .also we have other encapsulation like mpls ,gre that has their sizes that need to be put into consideration. Thanks Bombia and Chris .
Yes, more content with Chris and Wireshark. The real world examples are great and really helps show the thought process for going through the capture. It is also great how you go back and forth and ask questions that we the viewer may ask.
// MENU //
00:00 ▶ Coming Up
00:25 ▶ Intro
00:32 ▶ Chris introduction
00:47 ▶ Topic: Maximum Segment Size (MSS)
01:27 ▶ Explaining Maximum Transmission Unit (MTU)
08:42 ▶ Interface layout
10:25 ▶ David Bombal "War Story"
12:00 ▶ Wireshark demo
13:26 ▶ Increasing the MTU on your device for larger connections
16:27 ▶ Difference between MTU and MSS
19:36 ▶ Wireshark demo (cont'd)
24:58 ▶ Using Path MTU Discovery
27:02 ▶ Ping and Wireshark demo
33:32 ▶ Cool trick for Mac system
35:08 ▶ TCP/MSS Clamping
38:21 ▶ Chris Greer "War Story"
51:09 ▶ What happens if you can't capture a server
55:08 ▶ MSS Adjustment commands
56:55 ▶ Tunnel Path MTU Discovery
57:40 ▶ Figuring out 1432
01:02:52 ▶ Conclusion
01:04:48 ▶ "Cool features" in Wireshark
Previous video: ua-cam.com/video/rmFX1V49K8U/v-deo.html
// Wireshark PCAP files //
MTU PCAP: github.com/packetpioneer/youtube/blob/main/PMTUD.pcapng
War Story PCAP Client: github.com/packetpioneer/youtube/blob/main/slowfile-clientside.pcapng
War Story PCAP Server: github.com/packetpioneer/youtube/blob/main/slowfile-serverside.pcapng
Special “Thumbs Up” and “Subscribe” PCAP: github.com/packetpioneer/youtube/blob/main/tcp-streamexport.pcapng
// GOOD READING //
Network Implications of PMTUD: www.ipspace.net/kb/Internet/PMTUD/30-network-implications.html
Path MTU Discovery: www.ipspace.net/kb/Internet/PMTUD/20-mtu-discovery.html
Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPsec: www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html
Configuring TCP MSS Adjustment: www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/16-9/configuration_guide/ip/b_169_ip_3650_cg/configuring_tcp_mss_adjustment.pdf
Ethernet MTU and TCP MSS Adjustment Concept for PPPoE Connections: www.cisco.com/c/en/us/support/docs/ip/transmission-control-protocol-tcp/200932-Ethernet-MTU-and-TCP-MSS-Adjustment-Conc.html
// MY STUFF //
www.amazon.com/shop/davidbombal
// David SOCIAL //
Discord: discord.com/invite/usKSyzb
Twitter: twitter.com/davidbombal
Instagram: instagram.com/davidbombal
LinkedIn: www.linkedin.com/in/davidbombal
Facebook: facebook.com/davidbombal.co
TikTok: tiktok.com/@davidbombal
UA-cam: ua-cam.com/users/davidbombal
// Chris SOCIAL //
Udemy course: davidbombal.wiki/chriswireshark
LinkedIn: www.linkedin.com/in/cgreer/
UA-cam: ua-cam.com/users/ChrisGreer
Twitter: twitter.com/packetpioneer
// VLAD SOCIAL //
Twitter: twitter.com/Packet_vlad
PMTUD Blog: www.packettrain.net/2016/09/21/pmtud-or-not/
Thanks Vladimir Gerasimov!
// SPONSORS //
Interested in sponsoring my videos? Reach out to my team here: sponsors@davidbombal.com
Please note that links listed may be affiliate links and provide me with a small percentage/kickback should you use them to purchase any of the items listed or recommended. Thank you for supporting me and this channel!
i just want to say thank you from the bottom of my heart lots of love david ... i am getting started in cyber security and i have everything i wanted here in your channe thanks for your effort ......
Great video! I had a client running a Cisco 1100 in their home office - everything was working perfectly except for Netflix and Disney+ which seemed very strange. After a fair bit of investigation the issue turned out to be the MSS. They were on a home internet plan using pppoe so had to use IP TCP ADJUST-MSS 1452 command to account for the 8 byte overhead of pppoe.
RF: Thank you for providing your own war story, complete with your winning battle command. Good on ya', mate!
.
I like that you explained the battle and the winning command so if somebody has this problem in the field they can also use your strategy to fix it. God bless you
Amazing job, guys ! We see a lot of information written on web but when we see a live demo it's much more interesting and captivating. I've recently found David and Chris's channels and can't stop watching. Keep doing wireshark analyses for network problems ! It's awesome !
This was a great video. It reminded me and issue that I was dealing with around 2 years ago with Aruba APs. Our client has setup like this. One local WLAN aruba controller on site Spain and one redundant controller which was on remote site in Sweden. Clients reported issue with slow wifi. I found out that clients that are connected to APs which are connected to remote controller experience slowness. Other users connected to APs which used local wlan controller had no issue. So we started live troubleshooting and see that ping was ok but there was real slowness when opening web page or doing skype call (user was green but with packet loss). We started to manipulate ping request with more bytes and then packet loss appeared. Later I found out that APs has set "do not fragment" bit by default and we did calculation with wan team that packet is too big. So on our side we turn on preemption which will move APs to local WLC if it is online and WAN team did their config to divide packets to smaller size. This was really great issue and I enjoyed it :)
Btw. redundat remote wlc served for several small sites. Usually medium and big sites had 2 local WLCs but because of fact that some customer sites were too small they had setup with one local and one remote wlc.
Wow. I'm indeed wowed. This is an impressive and exceptional delivery.
It's unbelievable to find this kind of quality content for free. I felt I was in a classroom. I would appreciate a deep dive on the entire TCP/IP and Ethernet header.
Thanks for putting this out.
David and Chris, you won't believe how many networking professionals have no understanding on MTU or MSS, so thank you so much for this!
Please more on TCP and ideally the next with some focus on TTL (You touched on it in this video "My TTL started at 64 and it's now 54, so I'm going through 10 Routers")
Great video, subscribed and liked and will check out Chris' UA-cam channel
27:02 ▶ Ping and Wireshark demo. This part made my day. Best ever explanation I found on this concept. Thanks to you both 👍👍
Anohter fantastic collaboration and learning time. And the 'War Story" shows a real-world example for us to dive into and understand. Thank you David and Chris!
Boys, this video was incredible! Chris, you did an amazing job of practically explaining this and David’s input was very helpful.
Thank you so much Chris and David. This video is a goldmine and I learnt a lot from it. Watching it over and over again, and it just keeps making more sense.
I love the way you both engaged us all. I have really struggled in and out to know mtu and mss in detail. Now, I’m all good and proud. Thanks for the video. Keep up the great work!
I love these interviews, thank you so much for what you do David :) your channel has really helped me learn and grow as a web developer!
Fantastic! Very happy to hear that Sara!
Nothing better than waking up, grabbing a cup of coffee and watch David and Chris talk about networking for an hour. Absolute fantastic content!
Brilliant video. thanks guys. An hour with you guys and I learnt more than when I spend a year with my manager.
1:06:37 Yes please let’s get into the weeds. Great content as always from you and Chris. Thanks so much. 👍
Great video guys, very explanatory and easy to comprehend, I confirm this TCP issues are nigthmares, having dealing with a similar problem months ago where one of the branch offices experience a low vpn throughput and fragmentation, the issue was caused by incorrect mss clamping value,
it took few days to figure out whats was causing it. Like Chris said, its easier to find out what's wrong when the path is broken rather that in this trickier issues.
Thanks for sharing your vastly knowledge
Guys, this stuff is awesome. More videos, please, and continue this great work you guys are doing.
These videos answer the question: what if Robin Williams was a network engineering vlogger? In all seriousness though, great content; appreciate the clear and real-world explanations!
Excellent topic!! Reminding me why a Tunnel wasn't created however both routers were able to ping each other. Adjust for the overhead data
you two are like ... superman meets batman, no fighting though, only starting Justice League right away. Thanks for a great video.
Wow! Thanks crystal clear for me! i had some cloudy areas on this topic and now its clear. Amazingly explained. Thanks again to both of you.
Excellent content, Thanks again for sharing this valuable knowledge specially for newbies like me. I've been working on learning and mastering the fundamentals and this material makes it possible. Thanks Again.
Great to hear that! You're welcome!
Hi David & Chris,
Thank you for sharing your experience and knowledge. Now I understand why MTU & MSS-adjust commands are used in the mGRE tunnels comes into play.
Thanks alot guys and i can't wait to see you twos in your next collaboration.
Cheers! 🇵🇬
This was an amazing session! The clarity of thought and concept is too good and helpful !! I owe you guys !😇
Videos with Chris and you will never get to long! Great to watch and thanks for sharing your knowlege
Thanks, David. This hour lesson had so much information that I had to watch in slow-mode and analyse it for about 7 days! true gold.
Thanks again and I am subscribing to Chris' channel as well.
Have a great day.
Such Chris-David videos are the best thing happened in 2022. Keep it up❤️
I so needed this! For those of us who have spent days to tshoot an MTU issue.
David combines real war story to explain the symptoms we will experience if we have MTU and MSS issue. that is cool. Connects the theory with real life. More war stories please, that makes technical video more interesting.
Awesome guys, you guys made my day..First time watching a long video of yours, I hope this will help a lot. Thanks, Chris and David for teaching new kinds of stuff in TS.
it's always a treat when chris comes over for one of these deep dive videos
thanks David for the great content. much appreciated
If anyone is confused when Chris is explaining about ping capping at 1472 bytes because you have the IP and ICMP headers on top, but wondering what happened to the TCP header that he was talking about earlier in the video and why that isn't subtracted, remember that ICMP is a completely different L4 protocol. Hence in this example we do not need to take into account the 20 byte TCP header, just the ICMP and IP headers.
Layer 4 - UDP, TCP, ICMP
New video with Chris! Yessss! Thank you David for keep bringing him back.
Thank you for the review. I ve seen this in the 2000 during ATM implementations and many tunnel and datalinks. it is a great video to review and or learn in a practical matter what you just setup as default as MTU 1500 in your routers. and as you described to know why datalinks are slow when pings show Ips are reachable
Great episode David and Chris. The way Chris explains the topics is just amazing! Can't wait for the sequel.
I know I’m late to the game but dang this was well explained. Enjoyed all of it! Thank you for making this video!
Wow, David! What a blast from the past! That Cisco MTU/MSS white paper you showed a clip of was my "go to" doc for many years for helping to explain fragmentation issues to customers and peers alike. I still have the link to it in my bookmarks, since you never know when MTU's are gonna raise their ugly heads! As a long-time CCIE (#1937 just over 26 years ago) your content is refreshingly good!
We need time with Chris to get learn more about the packets, that transmit everything ..Great knowledge on handling the payload in defined mtu
Man this is gold, I'd love to see that video about 10Gpbs and 100Gbps connections and MTU tweaking, I happen to work at a service provider and there are so many customers out there that complain about running speed test on 10Gbps links while using laptops and not getting to the right speeds, of course at times there are more issues in the middle that can produce such outcome but most of the time is just either speed and duplex or MTU on the PE side that may cause things of that nature, at least in my experience.
Having same problem as you, but commonly laptops are coming capped to 1Gbps NIC card, so even incrementing their MTU won't benefit as they won't be able to test above 1gbps.
Thank you both for this detailed video. Appreciate the time and effort both put into for this interview! At many places I was lost - if the header bytes should be added or not. Should watch this video at-least twice to comprehend the depth of knowledge that this video contains. Pls continue your interviews with Chris. Absolutely love them and learning a lot from them. Looking forward for more Wireshark deep dive videos.
Very educational and good for troubleshooting we need more of Chris
awww you guys are such a cool duo! enjoyed watching videos with you both
Really fantastic content. Thank you so much!
I had network technology and Cisco trainings in my education two times, but it was always boring theory an I thought, it's a hard way to go any deeper in this stuff. But your videos show me, that it's all about good coaches.
One point about the Don't Fragment flag. Windows only sets it on TCP, but Linux does it for everything, IIRC. This can lead to a situation with Windows where TCP works fine, but there may be problems with UDP. On the other hand, Linux will not experience the UDP problem. Also, Path MTU Discovery is mandatory with IPv6, as fragmentation is not allowed.
Excellent video, been running into this a lot lately in a number of scenarios when replacing branch routers, first thing that breaks is Eap-TLS for wireless, those certificates do not like fragmentation, so have had to adjust MTU in a bunch of places.
David and Chris .. dropping dope content like never before 🔥🔥🔥🔥💯💯💯
Thank you Faran!
this is teaching 2.0. You can read school books, learn all the commands and theories as much as you can but this is both fun ánd interesting AND you even learning more then ever.
Thanks guys
Great vid. Loving it! Keep taking us down, all the way to the bottom!
Thanks!
Wow, so much details and information. Amazing video. Thank you David
Loved the WarStories, loved the vid, keep em comming David & Chris
Thank you! Will do!
I love learning with you guys. Maybe there is something slow in my brain, but I need to watch these more than once.
Glad you are enjoying the content. This is a deep dive - so you may need to watch it a few times as we cover a lot of information.
Awesome video. Confirmed my beliefs and what u have seen. Would like to see a video on pmtud, with mtu, and miss discussion
Had this issue with MPLS WAN circuits. We ran GRE over the MPLS circuits to carry our own EIGRP routing traffic. We had to do MSS "clamping" to account for the extra overhead from our GRE tunnel. We added another MPLS provider circuit to our headend and had to do the same but with a lower value to those locations using that MPLS provider. All to do with them using space in the TCP header for VLAN/MPLS.
12:44 Important note. For perfection, system can receive maximum. The reason for that is to avoid fragmentation. As we know, nowadays lots of applications use DF (do not fragment) bit. So, this is the reason to use standard number 1460 bytes.
This is an awesome topic to deep dive into. Thank you for sharing this, definitely will go and subscribe to Chris' channel.
Real life TCP/IP, real life MTU, just what our society needs! 👏👏👏
Cool instead of just disabling ipv6 on a router I now know what the reason could be when pages aren't loading on those random clients that uses corporate VPNs. MTU settings, never knew thanks :) Valuable information as always!
Thanks guys for your generosity, you’re awesome!
Excellent content, Thank you both of you!! Love to learn more..Waiting for next.
Thank you for another great video, always a pleasure to watch 🖤
great video david ❤️
Thanks! Glad the video helped :)
I already have experienced this in a large prod. environmen network where too much packet drops were coming and finally we were able to figure out the root cause to be the MTU size.
Gr8 video.
Thank you and thank you for sharing Zia! MTU can be an absolute nightmare!
@@davidbombal It happened on a Fortigate firewall, and after 2 days of troubleshooting and banging our heads and involving their tech support, this was identified. But still, no one was still able to identify where the MTU was getting changed in the packets complete path.
I hope this video will be a good starting point for many.
Kudos, and please keep up the good work!
Also there's an open source tool called mturoute which will return you the max possible mtu on the complete packet route.
Great video guys. Something is throwing me off, around the 33:22 mark when you're doing a 1472 df ping and it's replying at 1480, but Wireshark is capturing only 1464 for data + 8 for ICMP. Aren't we missing 8 bytes somewhere?
I noticed this discrepancy too, so I did some research to identify the answer: There is an 8-byte 'Timestamp' field shown by Wireshark right above the 'Data' payload section. This 'Timestamp' field is technically part of the ICMP payload, but is distinguished from the random data portion of the ICMP payload that is shown inside the 'Data' payload section.
You can see this for yourself by reproducing the ping + capture test demonstrated in the video; as you select each field of the ICMP header within the 'Packet Details' panel, pay attention to the highlighted characters within the 'Packet Bytes' panel - each individual character in this panel represents a single byte. Alternatively, the status bar located at the bottom left of the Wireshark application window contains a field description that also includes the byte size of the field.
By performing this exercise you can manually add up 8 total bytes of header + 8 total bytes of the Timestamp payload data + the 1464 bytes of random payload data, which equals 1480. And then of course you just add the 20 bytes from the IP header to get the total Ethernet MTU size of 1500 bytes.
Wow Wow Wow
All my concepts are clear
Best ever video
Thank you so much 💝💝💝
You're welcome Abhi!
Great discussion and I learned about jumbo packets. Thanks David! I really enjoy your show.
You're welcome! Glad you enjoyed it!
I have a pretty basic doubt on Chris' war story, where one of the routers on the client side did not have mss clamping enabled. In such a setup, shouldnt the cisco router on the server side, which had mss clamping enabled, clamped the MSS on the SYN packet from the client as well? I understand that when MSS clamping is enabled on an interface, it applies for both outbound and inbound traffic. Is it not the case?
Great video! Learned a lot! Thank you David and Chris!
As always , great videos with a lot of explanation . Thank you for going into detail of TCP communication .
We need more of this. There's a lack of affordable and good Wireshark content/courses
Check the video description for Chris's new Wireshark course on Udemy. That ia both great and affordable.
you guys are amazing teachers, thanks for sharing all that for free
very interesting content: generally, to avoid fragmentation on some routers it's possible to use MSS clamping.
E.g., if the router has more than two interfaces, on Cisco IOS this is possible using bidirectionally "ip tcp adjust-mss" interface configuration command.
:-)
Thanks for a good video on this interesting topic. A quick comment though, at about 14:00 you seem to imply that you can/should change the local NIC MTU to some arbitrary higher value. However, that cannot work if not all switch ports and the router interface on the same VLAN also have a non-default higher MTU. Additionally, if you will not have the full path, end-to-end, with higher MTU then changing the local LAN will likely not improve much.
WOW ! Grest video, thank you both so much, I just want more 👍🏼
David and Chris are just amazing i've enjoyed every sec
Thank you for this video! Could you please explain the difference between MTU and the calculated window size? They both seem related to the largest size a machine can send but I don't understand the contrast.
Finally, after a longtime looking for a right video for interviews
It's taken too long :) But, hope you enjoy the video!
@@davidbombal Absolutely it was a fantastic video again.
Thanks (again) for this very interesting video.
Something is still unclear to me however : on the Chris explanation at @1:00:00, he's saying that the root-cause was because they configured clamping while it was not necessary (if we can send 1472 bytes ping, we should be able to send 1460 bytes TCP segments). However, in this case, why were those "large" segments dropped ? MSS Adjust just changes the MSS value on the TCP SYN packets, but it does not drops further packets received being "above" this MSS value, am I wrong ?
Think he went off on a tangent, but wasn't allowed to test further to check his theory, but agree there's something in between dropping the packets. I'm guessing the original clamp was put in for a reason, but perhaps the client side was a new or replacement router, and the MTU issue was forgotten about.
Probably the answer is because ICMP (8B header) runs on top of IP (20B header), so when you ping and set the packet size (ping -s 1472) you set only the ICMP payload not including ICMP header (IP[20B] + ICMP[8B] + ICMPpayload[1472] = 1500B). Just remembering that ICMP with do not fragment tests MTU end-to-end and doesn't care about MSS as MSS is TCP only. Then a ping with 1472 would do perfectly fine.
TCP is another thing as it adds 20B header on top of IP, so the math would be IP(20B) + TCP(20B). A safety rule of thumb for me would be to subtract from biggest successful ping payload size extra 12B for accomodating TCP. Example: if a ping -s 1472 to a destination was successful, the MSS adjustment would be 1472 - 12 (TCPheader-ICMPheader) = 1460.
Just remembering here that IP header and TCP header sizes could vary (20B to 60B). For the worst case calculation (IP[60B] + TCP[60B]) the math would be 1472B (biggest successful ping payload) - 40B IP(adding extra 40B to already existing 20B) - 52 TCP(removing the 8 bytes from ICMP header that were already considered in ping) = 1380B
Not considering in this calculation any extra encapsulation overhead as VXLAN, etc...
Hi David after i finish up my video about the pure basics for computers in my mission to join the IT teaching community i will watch the full intervieuw :)
Excellent topic - a subject that I often revisit, especially dealing with VPNs.
This was a very interesting convo. Well I guess I'm into networking now XD.
GOOD STUFF. LOVING ALL THE NETWORKING CONTENT
Superb! Really enjoyed this thanks guys 😎
Great Video David! Pretty awesome series for understanding TCP. Just one question though, are you planning on releasing more videos in the series? These videos are a great help and would love to know if we have more coming.
Anyways, thanks for all the content. Cheers :)
Right time to sort my router out once and for all. Great video
the bridge explanation is so vivid :D
Very nice!! I love it . Thanks for all the great content 💎
I ran into these issues while using Aruba 8 Access Points traveling over Meraki VPN back to a controller via GRE. Had to reduce the AP MTU down to 1360 to stop the fragmentation.
Other tubers would tell you about MTU ping & fragmentation, just crank is down or make it higher. But none explain like chris ❤️
Networking is back 😀❤️
Networking will never go away :)
Best explanation heard ever. Thanks a lot.
Hi Gents…I need some help in further clarifying this ping with 1472 df.
1) My understanding is that the specified size in the ping command is the actual payload (icmp header + data = 8+1464 = 1472),but without IP header.But where are then those another 8 bytes not accounted for? (basically I see it as 1492,not 1500 )
2) When pinging from a router instead,,the size is actually including the IP Header.For example,a ping size of 1500 will give you a Icmp Data (1472) + Icmp H (8) + IP (20) = 1500 ,with 1514 length captured (plus L2)
So then…is this behavior regarding the ping size different between host/servers and routers?
Great video, but one thing doesn't make sense in the WAR story. Just because MSS was adjusted on one side wouldn't necessarily cause an issue, correct? Wouldn't that right hand router also have to have lowered it's MTU?
Thank you David and Chris!
I’ve always wondered how this worked! Thanks!
Hope you enjoy the video :)
So with a slow network situation as the example in the video.... Would you be able to test if a MTU change would work by setting the local device MTU in the network settings? I'm just thinking that the initial SYN packet should then contain a lower value so the server would try smaller packets immediately. Or am I missing something here.
David, I clicked a thumb up before the start of the vidoe
Thank you very much!
Firewalls that NAT also contribute if the bottleneck is after the firewall. The ICMP packet that comes back comes from a different IP than the destination and therefore gets dropped by the firewall ruleset. And because of the NAT, there is no way to figure out the connection it should go back to.
A video for all networks engr . Weldone guys
Thank you!