From d5b42174bd01367cc151d2f28de5ad765f356089 Mon Sep 17 00:00:00 2001
From: Pavel Raiskup <praiskup@redhat.com>
Date: Mar 08 2017 15:53:26 +0000
Subject: [backend] handle paramiko.SSHException in _run_ssh_cmd


Sometimes even when the VM allocation succeeds (spawn_playbook),
it might happen that ssh fails to respond a few moments later.

Not handling this properly means that the exception even leaks out
from do_job() call -> which means that (a) frontend is not
informed about build failure and (b) the worker/builder might not
be deallocated.

I observed similar issues when we were using ansible python API;
previously we did:

    self.run_ansible_with_check("/bin/rpm -q mock rsync")

.. with unhandled VmError from check_for_ans_error().  This was
replaced by:

    self._run_ssh_cmd("/bin/rpm -q mock rsync")

Without handling paramiko.SSHException however -- so I suppose the
problem is still there and thus fixing it.

---

diff --git a/backend/backend/mockremote/builder.py b/backend/backend/mockremote/builder.py
index 558a1c1..dead728 100644
--- a/backend/backend/mockremote/builder.py
+++ b/backend/backend/mockremote/builder.py
@@ -109,7 +109,12 @@ class Builder(object):
 
         self.log.info("BUILDER CMD: "+cmd)
 
-        stdin, stdout, stderr = conn.exec_command(cmd)
+        try:
+            stdin, stdout, stderr = conn.exec_command(cmd)
+        except paramiko.SSHException as err:
+            raise RemoteCmdError("Paramiko failure.",
+                                 cmd, -1, as_root, str(err), "(none)")
+
         rc = stdout.channel.recv_exit_status() # blocks
         out, err = stdout.read(), stderr.read()