Jumbo Frames Considerations

I have been doing some research and testing, and have made the following
discoveries:

1) Jumbo frames are not necessarily enabled on all switches and routers
2) Above fact doesn't mean anything is broken. Lowest common denominator
is MTU of 1500. 99% of hosts are set to 1500 anyway.
3) Any TCP session will use the mechanism called MSS (Maximum Segment
Size) to determine how big the packet size can be. It sends segments,
packaged into packets, which in turn gets packaged into frames (remember
studying for CCNA? TCP=Segment-->IP=Packet-->Ethernet=Frame). This could
result in dropped traffic only if both hosts are Jumbo frame enabled (they
would tell each other they are capable of sending large segments), but
somewhere in the path there is a link that is MTU 1500. When this happens,
that device is *supposed* to send an ICMP Need Fragmentation message, but
that doesn't always happen.

Consider the following examples:

1)
Host-----Switch---Host
1500 1500 1500
No problems here

2)
Host-----Switch---Host
9000 1500 1500
No problem here because TCP from the MTU 1500 host will tell the other
host to send smaller segments

3)
Host-----Switch---Host
1500 9216 1500
No problems here

4)
Host-----Switch---Host
9000 9216 1500
No Problem - same as option 2

5)
Host-----Switch---Host
9000 1500 9000
Umm, this could be a problem. The switch is supposed to send ICMP Need
Fragment message to the sending host if it sees jumbo frames. I hope that
the upper layer protocol is smart enough to know that when there are no
responses to change the segment size in TCP... Basically this scenario
forces me to double check all interfaces and vlans in the environment to
ensure they support Jumbo frames, or guarantee that ICMP Need Fragment
messages will be sent.

All that being said, I enabled jumbo frames (MTU 9000) on two VMs on the
same host and here is how we test:

Ping -f -l 8950 hostname

Result will show:
Reply from 192.168.1.10: bytes=8950 time=79ms TTL=128
Reply from 192.168.1.10: bytes=8950 time=1ms TTL=128
Reply from 192.168.1.10: bytes=8950 time=1ms TTL=128
Reply from 192.168.1.10: bytes=8950 time=1ms TTL=128

If for some reason jumbo frames are not allowed somewhere in the path, you
will either get

Packet needs to be fragmented but DF set.

Or

Request timed out.