Screen Shot 2016-04-22 at 12.44.38 PM

Issue

Three days after spinning up my first Cisco VIRL VM on VMWare Fusion, VM Maestro prompted me with the following alerts:

“Failed to collect current salt contact status”

I logged into UWM, navigated to VIRL Server > Salt Configuration and Status, and confirmed that indeed, my local VIRL VM was not authenticating with the Cisco salt masters.

To confirm if you’re having the same issue, open a terminal and run the following command:

virl@virl:~$ sudo salt-minion -l debug

Your output should assimilate the following:

[INFO ] Got list of available master addresses: [‘salt-master.cisco.com’, ‘salt-master-2.cisco.com’]
[DEBUG ] Attempting to authenticate with the Salt Master at 173.39.224.116
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO ] Master salt-master-2.cisco.com could not be reached, trying next master
[WARNING ] Master ip address changed from 173.39.224.116 to 173.39.236.96
[DEBUG ] Attempting to authenticate with the Salt Master at 173.39.236.96
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO ] Master salt-master.cisco.com could not be reached, trying next master (if any)
[ERROR ] No master could be reached or all masters denied the minions connection attempt.
[WARNING ] Stopping the Salt Minion

Diagnosis

Ensure basic network connectivity. Ping the configured salt masters.

virl@virl:~$ ping salt-master.cisco.com
PING salt-master.cisco.com (128.107.0.207) 56(84) bytes of data.
64 bytes from salt-master.cisco.com (128.107.0.207): icmp_seq=1 ttl=128 time=54.9 ms
64 bytes from salt-master.cisco.com (128.107.0.207): icmp_seq=2 ttl=128 time=245 ms
64 bytes from salt-master.cisco.com (128.107.0.207): icmp_seq=3 ttl=128 time=249 ms
64 bytes from salt-master.cisco.com (128.107.0.207): icmp_seq=4 ttl=128 time=259 ms
^C
— salt-master.cisco.com ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 54.919/202.397/259.720/85.308 m

or

virl@virl:~$ sudo salt-call -l debug test.ping

local:
True

Ensure the required TCP ports are opened.

virl@virl:~$ nc -zv salt-master.cisco.com 4505-4506
Connection to salt-master.cisco.com 4505 port [tcp/*] succeeded!
Connection to salt-master.cisco.com 4506 port [tcp/*] succeeded!

Ensure your license ID is configured correctly. It should match the license key that Cisco provided.

virl@virl:~$ sudo salt-call –local grains.get id
local:
xxxxxxxx.virl.info

Verify NTP is configured correctly. Authentication cannot occur if either the hypervisor or VM’s time is out of sync.

virl@virl:~$ ntpq -p
{Ensure this does not return INIT.}

Resolution

The issue for me turned out to be a hiccup with NTP. I fixed the issue as follows:

Ensure that the VM is configured to use valid NTP servers.

virl@virl:~$ less /etc/ntp.conf

Restart the services.

virl@virl:~$ sudo service ntp stop
virl@virl:~$ sudo ntpd -gq
virl@virl:~$ sudo service ntp start

This post goes into further detail for NTP troubleshooting.

Kudos to the people over at the /dev/innovate forums for providing excellent documentation to quickly diagnose & solve my issue!