vagrant up : ssh error during Ansible

[eouti]

Hello,
I'm trying to set up the default devstack environment you provide, using Vagrant and Ansible.
I clone the repo, and type vagrant up, but after creating the 3 VMs, installing the guest additions and all, i get the error below. Here's a tail of the log :
Is it a problem with my network config on the host ?
I'm running fresh ubuntu 14.04, eth0, regular config.

TASK: [cacher | start the devpi service] **************************************
<192.168.99.99> ESTABLISH CONNECTION FOR USER: vagrant
<192.168.99.99> REMOTE_MODULE service name=devpi state=restarted
<192.168.99.99> EXEC ['ssh', '-C', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/eoutin/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'IdentityFile=/home/eoutin/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.99.99', u'/bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=qgucjuiheurccrbiitoisubtkhhmugno] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-qgucjuiheurccrbiitoisubtkhhmugno; LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 /usr/bin/python'"'"''']
changed: [apt-pip-cache] => {"changed": true, "item": "", "name": "devpi", "state": "started"}

TASK: [cacher | install cronjob for restarts] *********************************
<192.168.99.99> ESTABLISH CONNECTION FOR USER: vagrant
<192.168.99.99> REMOTE_MODULE cron name="devpi" job="service devpi start > /dev/null"
<192.168.99.99> EXEC ['ssh', '-C', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/eoutin/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'IdentityFile=/home/eoutin/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.99.99', u'/bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=dgqwdiumpidmrjdsisodjeuvittteief] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-dgqwdiumpidmrjdsisodjeuvittteief; LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 /usr/bin/python'"'"''']
changed: [apt-pip-cache] => {"changed": true, "item": "", "jobs": ["devpi"]}

NOTIFIED: [cacher | restart server] *******************************************
<192.168.99.99> ESTABLISH CONNECTION FOR USER: vagrant
<192.168.99.99> REMOTE_MODULE command /sbin/reboot
<192.168.99.99> EXEC ['ssh', '-C', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/eoutin/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'IdentityFile=/home/eoutin/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.99.99', u'/bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=vosekamvfzwtxcujhqmrewkxrxnasxqz] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-vosekamvfzwtxcujhqmrewkxrxnasxqz; LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 /usr/bin/python'"'"''']
changed: [apt-pip-cache] => {"changed": true, "cmd": ["/sbin/reboot"], "delta": "0:00:00.022779", "end": "2014-06-08 23:24:16.636893", "item": "", "rc": 0, "start": "2014-06-08 23:24:16.614114", "stderr": "", "stdout": ""}

PLAY [waiting for caches to come online] **************************************
skipping: no hosts matched

PLAY [basic configs on all servers] *******************************************

GATHERING FACTS ***************************************************************
<192.168.99.99> ESTABLISH CONNECTION FOR USER: vagrant
<192.168.99.99> REMOTE_MODULE setup
<192.168.99.99> EXEC ['ssh', '-C', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/eoutin/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'IdentityFile=/home/eoutin/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.99.99', u'/bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=igahlzzfculmqadnemuctycfskwxpmqq] password: " -u root /bin/sh -c '"'"'echo SUDO-SUCCESS-igahlzzfculmqadnemuctycfskwxpmqq; LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 /usr/bin/python'"'"''']
fatal: [apt-pip-cache] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh

TASK: [common | disable mlocate runs] *****************************************
FATAL: no hosts matched or all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/home/eoutin/site.retry

apt-pip-cache : ok=17 changed=13 unreachable=1 failed=0

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Thanks for sharing your project, any help apreciated.
eo

[juergenbrendel]

Hello eo!

Thank you for the feedback. Under some rare circumstances I have found that a Vagrant created host may not be immediately reachable via SSH, resulting in Ansible failures. So, a retry can sometimes help under those circumstances.

However, I have noticed something else:

 PLAY [waiting for caches to come online] **************************************
 skipping: no hosts matched

That is strange. It's almost as if there are no hosts defined that should be waiting for the cache, which is unusual.

Could you tell me whether you have modified the hosts file ("vagrant_hosts_multi") or "Vagrantfile" in any way?

Furthermore, could you tell me how and when you installed the guest additions in the VMs? I normally don't bother with them at all and let the entire thing run through on its own, without any intervention from me.

Juergen

[eouti]

Thanks for your reply.
When i re-run "vagrant up", I get :
$ vagrant up
Bringing machine 'controller' up with 'virtualbox' provider...
Bringing machine 'compute-1' up with 'virtualbox' provider...
Bringing machine 'apt-pip-cache' up with 'virtualbox' provider...
==> controller: VirtualBox VM is already running.
==> compute-1: VirtualBox VM is already running.
==> apt-pip-cache: VirtualBox VM is already running.

Like if everything was ok.

No i didn't change anything after cloning the repo.
And the guest additions are installed by automatically, i don't do anythinh specific.

Cheers,
eo.

[juergenbrendel]

Once the machines are up and running, Vagrant won't initiate the "provisioning" step again. In other words, Ansible won't be run again just merely by doing a "vagrant up".

To force the running of the provisioning step (running Ansible again), use the command "vagrant provision".

Let's see what the output will be.

[eouti]

After the fail due to ssh error, i run "vagrant provision", and i end up with another error.
It seems to go further than the previous error.
I get :
TASK: [common | write PIP cache config for stack user] ************************
<192.168.99.99> ESTABLISH CONNECTION FOR USER: vagrant
fatal: [apt-pip-cache] => One or more undefined variables: 'FILES' is undefined

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/home/eoutin/site.retry

apt-pip-cache : ok=21 changed=6 unreachable=1 failed=0

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Did i miss something? I had a look to your code, but real newbie with Vagrant & Ansible.
Thank you for your help.

[juergenbrendel]

Hello!

Just an update: I'm tracing the issue you have reported and are working on a solution, which I hope to have available soon. I'll post here once I have an update.

Thank you for your feedback!

[juergenbrendel]

Hello eo,

I have just checked in some minor changes, which should fix the issue you have reported. Could you please check out the latest version and try once more? It would probably be best to do a "vagrant destroy" to get a completely fresh start, then try "vagrant up".

Please let me know how it works now.

[eouti]

Yea sure.
I updated the clone of your repo and started over.
I run "vagrant up", still interrupted by the ssh error. Then, i run the "vagrant provision" and the whole process finishes (even though i had a 2nd ssh error, re-run "vagrant provision")
But when i "vagrant ssh" into compute-1 or controller, the is nothing about devstack. The vagrant home directory doesn't have any of the folder you mentioned.
Seems like in my install, no hosts detected for compute or controller.
Should i edit vagrant_hosts_multi ? I use the default you provide.

Here's the tail of the "vagrant provision", finishing without error :

TASK: [common | updating the system] ******************************************
ok: [apt-pip-cache] => {"changed": false, "item": ""}

TASK: [common | reinstalling the language pack] *******************************
changed: [apt-pip-cache] => {"changed": true, "cmd": ["apt-get", "-y", "install", "--reinstall", "language-pack-en"], "delta": "0:00:01.381906", "end": "2014-06-10 18:27:34.899727", "item": "", "rc": 0, "start": "2014-06-10 18:27:33.517821", "stderr": "perl: warning: Setting locale failed.\nperl: warning: Please check that your locale settings:\n\tLANGUAGE = (unset),\n\tLC_ALL = (unset),\n\tLC_TIME = "fr_FR.UTF-8",\n\tLC_MONETARY = "fr_FR.UTF-8",\n\tLC_CTYPE = "C",\n\tLC_ADDRESS = "fr_FR.UTF-8",\n\tLC_TELEPHONE = "fr_FR.UTF-8",\n\tLC_NAME = "fr_FR.UTF-8",\n\tLC_MEASUREMENT = "fr_FR.UTF-8",\n\tLC_IDENTIFICATION = "fr_FR.UTF-8",\n\tLC_NUMERIC = "fr_FR.UTF-8",\n\tLC_PAPER = "fr_FR.UTF-8",\n\tLANG = "C"\n are supported and installed on your system.\nperl: warning: Falling back to the standard locale ("C").\nlocale: Cannot set LC_ALL to default locale: No such file or directory\ndebconf: unable to initialize frontend: Dialog\ndebconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)\ndebconf: falling back to frontend: Readline\ndebconf: unable to initialize frontend: Readline\ndebconf: (This frontend requires a controlling tty.)\ndebconf: falling back to frontend: Teletype\ndpkg-preconfigure: unable to re-open stdin: ", "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libfontenc1 libgl1-mesa-dri\n libgl1-mesa-glx libglapi-mesa libice6 libllvm3.4 libpciaccess0 libpixman-1-0\n libsm6 libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7 libxcb-dri2-0 libxcb-dri3-0\n libxcb-glx0 libxcb-present0 libxcb-sync1 libxcomposite1 libxdamage1\n libxfixes3 libxfont1 libxkbfile1 libxmu6 libxpm4 libxrandr2 libxrender1\n libxshmfence1 libxt6 libxxf86vm1 x11-common x11-xkb-utils xfonts-base\n xfonts-encodings xfonts-utils xserver-common xserver-xorg-core\nUse 'apt-get autoremove' to remove them.\n0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.\nNeed to get 0 B/287 kB of archives.\nAfter this operation, 0 B of additional disk space will be used.\n(Reading database ... 89174 files and directories currently installed.)\nPreparing to unpack .../language-pack-en_1%3a14.04+20140522_all.deb ...\nUnpacking language-pack-en (1:14.04+20140522) over (1:14.04+20140522) ...\nSetting up language-pack-en (1:14.04+20140522) ..."}

PLAY [cluster] ****************************************************************
skipping: no hosts matched

PLAY [cluster] ****************************************************************
skipping: no hosts matched

PLAY [controller-hosts] *******************************************************
skipping: no hosts matched

PLAY [compute-hosts] **********************************************************
skipping: no hosts matched

PLAY [controller-hosts] *******************************************************
skipping: no hosts matched

PLAY RECAP ********************************************************************
apt-pip-cache : ok=25 changed=11 unreachable=0 failed=0

Thanks again for your help Juergen.
eo.

Edit : just to be more precise concerning software versions
Ansible : 1.6.2
Vagrant : 1.6.3
VirtualBox : 4.3.12
On Ubuntu 14.04 host, Linux 3.13.0-27-generic x64

[juergenbrendel]

So, your Vagrantfile as well as vagrant_hosts_multi file are exactly as they are in the repo? You didn't change either one of them, right?

[eouti]

Yes, unmodified both of them.

[juergenbrendel]

I just installed from scratch on a different computer and it all runs through without issues, aside from the occasional SSH errors. The SSH issues are annoying, but a simple restart of the 'provision' step usually takes care of that.

The "skipping: no hosts matched" messages you see indicate that there is another issue, that there is something different about your setup, but it's just not clear right now what it is.

Let me ask you this: Are there any other Vagrant environments you have running? For example, did you attempt to run this in some other directory, possibly still leaving some instances running? If so, could you please make sure to shut down all of those?

I see that you are using a very recent version of Vagrant, so I assume you manually installed it? I installed it on 14.04 via 'apt-get' and just got version 1.4.3. That's a difference, but somehow I don't think that's the reason here.

Could you please upload th entire Ansible output on a server somewhere, so that I could have a look?

[eouti]

I finally got this working :)
When starting over the build of the devstack-environment, I used to do "vagrant halt" to shutdown the VMs and then delete them graphically within the VirtualBox GUI. I even used to rm the working directory and reclone your repo, to re-start a fresh install from scratch.
When I read your last answer, I realized that the only residual thing from different installs is the initial Vagrant box. I decided to remove/purge Vagrant, reinstall it (version 1.4.3), so it doesn't have any box beforehand.
Running "vagrant up" now works like a charm, and I end up with a clean devstack-environment.
Thanks again,
cheers,

eo

[juergenbrendel]

That's great, I'm so glad to hear that! Thank you for sticking with it. Please let me know if there are any other questions or if you encounter any issues.