ESXi Network Troubleshooting Tools

In the previous post about the ESXi network IOchain we explored the various constructs that belong to the network path. This blog post builds on top of that and focuses on the tools for advanced network troubleshooting and verification. Today, vSphere ESXi is packaged with a extensive toolset that helps you to check connectivity or verify bandwidth availability. Some tools are not only applicable for inside your ESXi box, but also very usable for the physical network components involved in the network paths.

Access to the ESXi shell is a necessity as the commands are executed here. A good starting point for connectivity troubleshooting is the esxtop network view. Also, the esxcli network commandlet provides a lot of information. We also have (vmk)ping, traceroute at our disposal. However, if you are required to dig deeper into an network issue, the following list of tools might help you out:

  • net-stats
  • pktcap-uw
  • nc
  • iperf


We’ll start of with one of my favorites; net-stats. This command can get you a lot of deep dive insights on what is happening under the covers of networking on a ESXi host as it can collect port stats and . The command is quite extensive as it allows for a lot of options. The net-stats -h command displays all flags. The most common one being the list option. Use net-stats -l to determine the switchport numbers and MAC addresses for all VMkernel interfaces, vmnic uplinks and vNIC ports. This information is also used for input for other tools described in the blog post.

To give some more examples, net-stats can also provide in-depth details on what worldlets (or CPU threads, listed as “sys”) are spun up for handling network IO by issuing net-stats with the following flags: net-stats -A -t vW. Output provided by these options help in verifying if NetQueue or Receive Side Scaling (RSS) is active for vmnic’s by mapping the “sys” output to the worldlet name using i.e. the vsi shell (vsish -e cat /world/<world id>/name).

Using different options, net-stats provides great insights on network behaviour.


vSphere ESXi 5.5 and later is shipped with pktcap-uw next to the tcpdump-uw tool. The latter was already shipped with ESX(i) since the early versions. Both tools compliment each other and can be used together or separately. The tcpdump-uw tool is only able to capture packets at the VMkernel interface level. The pktcap-uw tool allows you to also capture frames at the uplink, vSwith or, virtual port levels. Therefor, is provides for a lot more flexibility and improved troubleshooting. We have a KB article that provides detailed information about different options using pktcap-uw.

I would like to highlight some of the options with regards to the diagram we know from the previous blog post. It details on what syntax to use to get the correct data of the network path level of choice.

Besides capturing packets on a specific point along the path, we do have more general options and tips & tricks. For example:

  • Filter all frames for a specific MAC address:
    pktcap-uw --mac xx:xx:xx:xx:xx:xx
  • Filter explicitly for a IP:
    pktcap-uw --ip x.x.x.x
  • Automatically run pktcap-uw by using sleep:
    pktcap-uw $sleep 120; pkill pktcap-uw
  • Limit your capture by packet count:
    pktcap-uw -c 100
  • Output your capture to review offline:
    pktcap-uw -P -o /tmp/example.pcap


Next up is perhaps a less known command. NC stands for NetCat and is a oldschool linux command. It is the perfect tool for verifying connectivity on a specific port since telnet is not available in ESXi. For example, to verify if the iSCSI port on a storage array is reachable.

nc -z <destination IP> 3260

Please review this KB article for more information!


Iperf is a bandwidth tool. It tests the maximum bandwidth between a source and destination. Typically used from within a VM but in this case the source and destination being the ESXi host itself. It is baked into ESXi because of the VMware vSAN proactive network performance test that is accessible in the UI. We can ‘mis-use’ iperf for other purposes as well. We can use any VMkernel interface in a ESXi host.

By default, ESXi will not let you run iperf3 in server mode from the shell. It raises an “Operation not permitted” error. The trick is to create a copy of iperf using the following syntax:

cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf3.copy

Because by default iperf3 runs on ports that are not permitted in the ESXi firewall. One option is to temporarily disable the firewall. Another option is to create additional ESXi firewall entries. Just be sure to enable the ESXi firewall afterwards!

esxcli network firewall set --enabled <false / true>

Now issue iperf3.copy with the -s tag on the destination ESXi host. The beauty of this, is that all VMkernel IP’s can be used. So if you want to test the management network, bind iperf3 with the management IP. The same goes for the vMotion network or any other VMkernel interface!

Server (ESXi host 1):

/usr/lib/vmware/vsan/bin/iperf3.copy -s -B

Client (ESXi host 2):

/usr/lib/vmware/vsan/bin/iperf3.copy -c

Output will look similar to the following screenshot:

All other iperf3 options are supported, like parallelism, format or the maximum segment size.

To Conclude…

This concludes the write-up on ESXi network troubleshooting. The tools discussed are really extensive in number of options and settings, we only scratched the surface here. I encourage you to get familiar with all tools available. Explore and test them to get better understanding on virtual network characteristics in your VMware environment.


The post ESXi Network Troubleshooting Tools appeared first on VMware vSphere Blog.

10 inspirational quotes than can improve yourself

It might take a little coffee or probably a few rounds of beer or any other booze you could get your hands on when it comes to relaxing after a hard day’s work. Well, yeah I’m guilty about that one as well, unless I’m caught dead wearing a lampshade over my head after a few rounds of vodka… half-naked! Okay, bad example and I apologize to everyone reading this after getting nightmares about me in that state of drunken stupor.

Just don’t ask how it happened, please.

But what’s really interesting is that how do people go through the usual part of life when faced with vein-popping stress? I mean, the new age thing like Zen or yoga is one of the good things and it actually works. Is there room for the intellectual side of people who can actually smell the roses-in-a-can while on the move? It kind of had me thinking that there really must be something in this ‘mind-over-matter’ thing.

Humor is indeed the best medicine there is whenever you are. I mean anyone can pay good money to listen to a comedian just to make you wet your pants after laughing so hard. Despite of what’s been happening, and to those who has gone though the ordeal, it’s better to just laugh while facing the troubles with a clear mind than anger with a clouded vision. One of my favorite celebrities of all time may have to be Woody Allen. Now this is one guy who gives you the in-your-face bluntness that he pulls out with gusto, even without even trying. You can talk just about anything with a man, and he’s bound to mock the subject and you’ll end up laughing rather than being upset about it.

Woody Allen has this to say:

1. “Money is better than poverty, if only for financial reasons.” It sounds good to me, I mean the practicality of all things does involve money but it doesn’t have to take an arm and a leg to get it.

2. “I believe there is something out there watching us. Unfortunately, it’s the government.” ‘Nuff said.

3. “There are worse things in life than death. Have you ever spent an evening with an insurance salesman?” This happens to be one of the classic ones. I mean the issue about life’s little problems isn’t all that bad, until ‘he’ shows up.

Sure, relationships can get complicated, or does have its complications that probably any author about relationships is bound to discover it soon. We follow what our heart desires, unless you’re talking about the heart as in the heart that pump blood throughout your body.

4. “Love is the answer, but while you’re waiting for the answer, sex raises some pretty interesting questions.” And if you want more, just keep on asking!

5. “A fast word about oral contraception. I asked a girl to go to bed with me, she said ‘no’.” It sounds, ‘practical’, I think.

And when it comes to everyday life, he really knows how to make the best out of every possible scenario, and it doesn’t involve a lawsuit if he strikes a nerve.

6. “Basically my wife was immature. I’d be at home in the bath and she’d come in and sink my boats.” I never had a boat in my bathtub before. Just staring at it while soaking in hot water makes me seasick already.

7. “I am not afraid of death, I just don’t want to be there when it happens.” If it rains, it pours.

8. “I am thankful for laughter, except when milk comes out of my nose.” It could get worse when you’re guzzling on beer… or mouthwash, and it happened to me once!

9. “If you want to make God laugh, tell him about your plans.” At least he doesn’t smite us with lightning, and I’m thankful for that.

And despite of what may happen to all of us in the next ten, twenty, or even thirty years, I guess we all have to see things in a different kind of light and not just perspective. I can’t seem to imagine life without any piece of wisdom that could guide us. Whether we’re religious or not, it takes more courage to accept your fears and learn how to deal with them is all that matters when it comes to even just getting along.

And to sum things up, here is the last nugget of wisdom to go by… however, whenever, and wherever we may be.

10. “The talent for being happy is appreciating and liking what you have, instead of what you don’t have.”


Safety Emergency Preparation Tips When There is a Tornado

From March to May of every year, the southern states are occasionally plagued by disastrous weather phenomenon known as tornadoes. For northern states, the risk comes during spring to the onset of summer.

Tornadoes are considered as storms. And you know that no storm is not violent. The dangers caused by tornadoes are not due to the gustiness of the winds but by sudden and unanticipated occurrence.

Unlike hurricanes, which are comparatively more powerful, the onset of a tornado can not be easily tracked. Satellite weather systems can not easily spot them because of their sudden development and their relatively smaller scale.

Thus, people can only tell that there is a tornado when they actually see one. Tornadoes are massive and destructive mass of air accumulating rapidly is a circular manner. The wind is so strong that almost everything that comes to the way of the tornado is easily flown into it.

How to tell if a tornado is approaching

Before any safety measure, it is imperative that you be knowledgeable about the appearance and the indicators of the presence of the violent weather phenomenon.

Take note that before a tornado hits an area, the wind may not be destructive minutes before the tornado hits the place. Often, before a tornado is observed, the wind may be very still. There may or may not be presence of rain.

Usually, tornadoes occur at the trailing end of a thunderstorm. Thus, if there is strong thunderstorm, be prepared on the possibility that a tornado may come anytime during of before the thunderstorm ends.

Because tornadoes are made up basically of strong and whirling winds, one can not easily see them. Often, reported cases indicate that the tornado appears like a cloud-like structure.

Tornadoes can be seen due to the debris and dust they have accumulated while tracking their courses. You may see the debris rotating in the air in circular, whirling motion.

Before a tornado hits an area, you can also hear loud roaring sound that is almost similar to an approaching freight train. The wind will also become turbulent and destructive.

Safety measures before a tornado

One indicator that a tornado will possibly hit an area is dark and often violet to greenish sky, which is prevalent during destructive thunderstorms.

When you see and feel the indications of an approaching tornado, find a safe shelter right away. Remember, tornadoes are destructive and fatal not merely because of the strong winds but because of the debris they have accumulated.

These debris vary from roof, to large rocks, to large chunks of trees. When people are hit by such debris, that is the time the tornado becomes fatal. So make sure the shelter you find would protect you from any such debris.

Safety measures during a tornado

Homes located in areas that are often hit by tornadoes almost always have basements. Basements offer reliable protection to people during tornadoes, so find one during an attack of a tornado.

If you are in your car, just get out and find a safe shelter. Or simply lie down on sturdy structure. Lie flat and cover the head with hands. Be alert and look out for falling debris coming from the tornado.

After the tornado, make sure any structure in your home is not significantly damaged. Otherwise, seek professional help all at once to ensure your safety.

Tornadoes are phenomenal that one can not prevent it from occurring. However, if you know the how-to’s of being safe during such weather disturbances, you would avoid being one of the many casualties arising from such a weather turbulence.

Emergency Preparation Tips for Typhoons

Almost all countries near major ocean bodies are plagued occasionally by the occurrence of typhoons. In the Pacific, the weather disturbance is widely called typhoon. It is cyclone in the Indian ocean and in the United States, where most often it comes from the Atlantic, it is called hurricane.

A typhoon is a massive accumulation of destructive winds and torrential rains over a huge body of water, particularly ocean. The accumulation is facilitated by an area characterized by the presence of a low-pressure, which makes evaporation and wind development faster.

Typhoons do not develop overnight. Often, it takes about a few days before one is developed. Usually, typhoons start as thunderstorms that accumulate strength and intensify into a massive weather disturbance.

Typhoons are easily tracked and monitored by weather satellites because of their huge radius and accumulated cloud system.

Thus, weather bureaus are always able to track a development of one, making it possible for concerned government and safety bodies to release typhoon warnings for the safety of people who might be affected by the weather onset.

A typhoon is an almost predictable calamity. Almost, because its tracks can easily change, but certain, because its presence is easily monitored.

Preparing for a typhoon

Countries that are always visited by typhoons have instituted safety measures to prepare people for the destructive weather occurrence and to help curb, if not totally prevent, injuries and casualties.

When a typhoon is about to hit an area, typhoon warnings should already be released, at least 24 to 48 hours before the expected landfall of the weather disturbance.

Thus, people, whose residences are within the warning areas, should first and foremost make the necessary safety precautions. Here are some of them.

· Store enough food and drinking water that would last for a few days. The typhoons may bring in torrential rains and destructive winds that may ruin infrastructure. Groceries and markets might not be able to resume operations after a typhoon, so make sure your supplies would last until the situation goes back to normal.
· Make sure your home is sturdy enough to endure potential gusty winds and flash floods. If not, just evacuate and go to designated safe evacuation or safety center.
· During the typhoon, you may opt to put off electric currents within the house to prevent any electrical accidents. The local utility firm will voluntarily cut off the electric supply.
· Typhoons bring in rains and winds so make sure you have jackets and umbrellas, in case you may need to flee.
· If your home is safe enough, just stay indoors before and during the typhoon to ensure safety. During typhoons, electric wires might fall down to the streets and compromise safety. Same goes for falling, or even flying debris, which are the usual cause of injuries and casualties.
· Always monitor news updates regarding the typhoon. Get transistor radios to make sure there would be no news blackout in case the electric and communication facilities shut down.

Remember that typhoons are weather phenomenon. One can never prevent it from forming and coming over. However, what people can do is to help prevent the destruction of the weather disturbance.

They can help by not becoming one of the casualties. Life is too precious and it is not worth losing it due to some occasional weather occurrence like the typhoon.