Can't complete install - RabbitMQ exception throw

[01000101]

It's not very clear what exactly should be private vs. public IPs, so I chose localhost as private and the IP that agents would use for connection as the public IP.

Using the community edition pre-built RPM

cloudify-manager-install-17.12.14-community.x86_64.rpm

Operating System information is Red Hat Enterprise Linux Server release 7.4 (Maipo)

Firewall is disabled

$ sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2017-12-10 22:27:19 EST; 8min ago

SELinux is permissive

$ sudo sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Here's the install log + error message

$ cfy_manager install --private-ip 127.0.0.1 --public-ip 10.11.49.14
2017-12-10 22:31:08,994 - [MAIN] - NOTICE - Installing Cloudify Manager...
2017-12-10 22:31:08,994 - [VALIDATIONS] - NOTICE - Validating local machine...
2017-12-10 22:31:08,994 - [VALIDATIONS] - INFO - Validating Python version...
2017-12-10 22:31:08,995 - [VALIDATIONS] - INFO - Validating supported distributions...
2017-12-10 22:31:08,995 - [VALIDATIONS] - INFO - Validating memory requirement...
2017-12-10 22:31:08,995 - [VALIDATIONS] - INFO - Validating disk space requirement...
2017-12-10 22:31:08,998 - [VALIDATIONS] - INFO - Validating OpenSSL version...
2017-12-10 22:31:09,024 - [VALIDATIONS] - INFO - Validating package prerequisites...
2017-12-10 22:31:09,114 - [VALIDATIONS] - INFO - Validating user `jcornutt-ctr` has sudo permissions...
2017-12-10 22:31:09,153 - [VALIDATIONS] - NOTICE - All validations passed successfully!
2017-12-10 22:31:09,153 - [GLOBALS] - INFO - SSL is disabled, setting rest port to 80 and rest protocols to http...
2017-12-10 22:31:09,153 - [GLOBALS] - INFO - Generated password: cE6P0xtkPgyY
2017-12-10 22:31:09,153 - [MANAGER] - NOTICE - Installing Cloudify Manager resources...
2017-12-10 22:31:09,153 - [MANAGER] - INFO - Copying agent packages...
2017-12-10 22:31:09,894 - [USERS] - INFO - Creating group cfyuser
2017-12-10 22:31:09,952 - [USERS] - INFO - Creating user cfyuser, home: /etc/cloudify...
2017-12-10 22:31:10,517 - [MANAGER] - NOTICE - Cloudify Manager resources successfully installed
2017-12-10 22:31:10,518 - [MANAGER-IP-SETTER] - NOTICE - Installing Manager IP Setter...
2017-12-10 22:31:10,772 - [SUDOERS] - INFO - Allowing user `cfyuser` to run `/opt/cloudify/manager-ip-setter/update-provider-context.py`
2017-12-10 22:31:11,130 - [SUDOERS] - INFO - Allowing user `cfyuser` to run `/opt/cloudify/manager-ip-setter/manager-ip-setter.sh`
2017-12-10 22:31:11,382 - [MANAGER-IP-SETTER] - INFO - Set manager ip on boot is disabled.
2017-12-10 22:31:11,383 - [MANAGER-IP-SETTER] - NOTICE - Manager IP Setter successfully installed
2017-12-10 22:31:11,383 - [NGINX] - NOTICE - Installing NGINX...
2017-12-10 22:31:11,515 - [YUM] - INFO - Installing nginx...
2017-12-10 22:31:13,417 - [NGINX] - INFO - Handling CA certificate...
2017-12-10 22:31:13,418 - [NGINX] - INFO - Generating CA certificate...
2017-12-10 22:31:13,766 - [NGINX] - INFO - Handling internal certificate...
2017-12-10 22:31:13,766 - [NGINX] - INFO - Generating internal certificate...
2017-12-10 22:31:14,429 - [NGINX] - INFO - Handling external certificate...
2017-12-10 22:31:14,429 - [NGINX] - INFO - Generating external certificate...
2017-12-10 22:31:15,060 - [NGINX] - INFO - Deploying Nginx configuration files...
2017-12-10 22:31:16,496 - [NGINX] - INFO - Starting NGINX service...
2017-12-10 22:31:16,772 - [NGINX] - NOTICE - NGINX successfully installed
2017-12-10 22:31:16,772 - [PYTHON] - NOTICE - Installing Python dependencies...
2017-12-10 22:31:16,897 - [PYTHON] - INFO - Validating pip installation...
2017-12-10 22:31:17,144 - [PYTHON] - NOTICE - Python dependencies successfully installed
2017-12-10 22:31:17,145 - [POSTGRESQL] - NOTICE - Installing PostgreSQL...
2017-12-10 22:31:17,366 - [YUM] - INFO - Installing postgresql95-libs...
2017-12-10 22:31:18,603 - [YUM] - INFO - Installing postgresql95...
2017-12-10 22:31:20,160 - [YUM] - INFO - Installing postgresql95-contrib...
2017-12-10 22:31:21,510 - [YUM] - INFO - Installing postgresql95-server...
2017-12-10 22:31:23,701 - [YUM] - INFO - Installing postgresql95-devel...
2017-12-10 22:31:25,551 - [YUM] - INFO - Installing python-psycopg2...
2017-12-10 22:31:30,085 - [POSTGRESQL] - INFO - Starting PostgreSQL service...
2017-12-10 22:31:32,271 - [POSTGRESQL] - INFO - Updating PostgreSQL configuration...
2017-12-10 22:31:35,025 - [POSTGRESQL] - INFO - Creating default PostgreSQL DB: cloudify_db...
2017-12-10 22:31:36,301 - [POSTGRESQL] - NOTICE - PostgreSQL successfully installed
2017-12-10 22:31:36,302 - [RABBITMQ] - NOTICE - Installing RabbitMQ...
2017-12-10 22:31:36,415 - [YUM] - INFO - Installing erlang...
2017-12-10 22:31:38,540 - [YUM] - INFO - Installing rabbitmq-server...
2017-12-10 22:31:41,089 - [RABBITMQ] - INFO - Initializing RabbitMQ...
2017-12-10 22:31:42,666 - [NETWORK] - INFO - Waiting for localhost:5672 to become available...
2017-12-10 22:31:42,670 - [NETWORK] - INFO - localhost:5672 is open!
2017-12-10 22:31:42,713 - [RABBITMQ] - INFO - Enabling RabbitMQ Plugins...
2017-12-10 22:31:44,177 - [RABBITMQ] - INFO - Disabling RabbitMQ guest user...
2017-12-10 22:31:45,002 - [RABBITMQ] - INFO - Creating new user and setting permissions...
2017-12-10 22:31:45,880 - [RABBITMQ] - INFO - Starting RabbitMQ Service...
2017-12-10 22:31:50,197 - [NETWORK] - INFO - Waiting for localhost:5671 to become available...
2017-12-10 22:31:50,199 - [NETWORK] - INFO - localhost:5671 is open!
2017-12-10 22:31:50,199 - [RABBITMQ] - INFO - Setting RabbitMQ Policies...
2017-12-10 22:31:51,213 - [MAIN] - ERROR - BootstrapError: Failed running command: sudo rabbitmqctl set_policy logs_queue_message_policy ^cloudify-logs$ {"message-ttl": 60000, "max-length": 1000000} --apply-to queues (Error: unable to connect to node 'cloudify-manager@localhost': nodedown

DIAGNOSTICS
===========

attempted to contact: ['cloudify-manager@localhost']

cloudify-manager@localhost:
  * connected to epmd (port 4369) on localhost
  * epmd reports: node 'cloudify-manager' not running at all
                  other nodes on localhost: ['rabbitmq-cli-8730']
  * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-8730@cfy-mgr-01'
- home dir: /var/lib/rabbitmq
- cookie hash: m60BDBAf1wQU2+K+REAySg==

).

[mcouthon]

Well, the public IP is the one accessible from outside the network, while the private IP (usually 192.168.X.X or 10.0.X.X or something like this) is the internal IP.
So you might want to try either providing one like this, or trying 127.0.0.1 instead of localhost.

What's more, you code look at /var/log/cloudify/manager/cfy_manager.log, and see if the DEBUG logs can shed more light on the issue.

[mcouthon]

@01000101 Is it working now?

[andrew-lee-1089]

We've just hit this on Cloudify V4.5. We weren't using localhost as private or public IP.

2019-02-06 16:10:10,545 - [RABBITMQ] - INFO - Initializing RabbitMQ...
2019-02-06 16:10:10,545 - [UTILS] - DEBUG - Removing /var/lib/rabbitmq/mnesia...
2019-02-06 16:10:10,546 - [UTILS] - DEBUG - Running: ['sudo', 'rm', '-rf', '/var/lib/rabbitmq/mnesia']
2019-02-06 16:10:10,557 - [UTILS] - DEBUG - Removing /etc/rabbitmq/rabbitmq.config...
2019-02-06 16:10:10,557 - [UTILS] - DEBUG - Running: ['sudo', 'rm', '-rf', '/etc/rabbitmq/rabbitmq.config']
2019-02-06 16:10:10,569 - [UTILS] - DEBUG - Running: ['sudo', 'systemctl', 'daemon-reload']
2019-02-06 16:10:10,608 - [UTILS] - DEBUG - Running: ['sudo', 'systemctl', 'restart', 'cloudify-rabbitmq']
2019-02-06 16:10:14,707 - [NETWORK] - INFO - Waiting for localhost:5671 to become available...
2019-02-06 16:10:14,710 - [NETWORK] - INFO - localhost:5671 is open!
2019-02-06 16:10:14,710 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'list_users']
2019-02-06 16:10:15,239 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'list_users']
2019-02-06 16:10:15,798 - [RABBITMQ] - INFO - Creating new user and setting permissions...
2019-02-06 16:10:15,799 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'add_user', 'cloudify', 'c10udify']
2019-02-06 16:10:16,341 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'set_permissions', 'cloudify', '.*', '.*', '.*']
2019-02-06 16:10:16,914 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'set_user_tags', 'cloudify', 'administrator']
2019-02-06 16:10:17,478 - [RABBITMQ] - INFO - Starting RabbitMQ Service...
2019-02-06 16:10:17,479 - [UTILS] - DEBUG - Running: ['sudo', 'systemctl', 'restart', 'cloudify-rabbitmq']
2019-02-06 16:10:26,163 - [NETWORK] - INFO - Waiting for localhost:5671 to become available...
2019-02-06 16:10:26,165 - [NETWORK] - INFO - localhost:5671 is open!
2019-02-06 16:10:26,165 - [RABBITMQ] - INFO - Setting RabbitMQ Policies...
2019-02-06 16:10:26,166 - [RABBITMQ] - DEBUG - Setting policy logs_queue_message_policy on queues ^cloudify-logs$ to {"message-ttl": 60000, "max-length": 1000000}
2019-02-06 16:10:26,166 - [UTILS] - DEBUG - Running: ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'set_policy', 'logs_queue_message_policy', '^cloudify-logs$', '{"message-ttl": 60000, "max-length": 1000000}', '--apply-to', 'queues']
2019-02-06 16:10:34,709 - [FILES] - DEBUG - Cleaning temporary files...
2019-02-06 16:10:34,709 - [UTILS] - DEBUG - Removing /tmp/tmpetoLJt...
2019-02-06 16:10:34,709 - [UTILS] - DEBUG - Running: ['sudo', 'rm', '-rf', '/tmp/tmpetoLJt']
2019-02-06 16:10:34,721 - [UTILS] - DEBUG - Removing /tmp/tmpSDwOW6...
2019-02-06 16:10:34,721 - [UTILS] - DEBUG - Running: ['sudo', 'rm', '-rf', '/tmp/tmpSDwOW6']
2019-02-06 16:10:34,732 - [UTILS] - DEBUG - Removing /tmp/tmpgybtFR...
2019-02-06 16:10:34,733 - [UTILS] - DEBUG - Running: ['sudo', 'rm', '-rf', '/tmp/tmpgybtFR']
2019-02-06 16:10:34,743 - [FILES] - DEBUG - Cleaned temporary files
2019-02-06 16:10:34,743 - [MAIN] - ERROR - BootstrapError: Failed running command: sudo rabbitmqctl -n cloudify-manager@localhost set_policy logs_queue_message_policy ^cloudify-logs$ {"message-ttl": 60000, "max-length": 1000000} --apply-to queues (Error: unable to perform an operation on node 'cloudify-manager@localhost'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node cloudify-manager@localhost

DIAGNOSTICS
===========

attempted to contact: ['cloudify-manager@localhost']

cloudify-manager@localhost:
  * connected to epmd (port 4369) on localhost
  * node cloudify-manager@localhost up, 'rabbit' application running

Current node details:
 * node name: 'rabbitmqcli17@metaswitch-commissioning-cfy'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: n5fzgvnnW0mW5gApc/ZRRg==

).
2019-02-06 16:10:34,747 - [MAIN] - DEBUG - Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 367, in execute
    self._wrap_coverage(self._wrap_profiling, self._execute)
  File ".bootstrap/_pex/pex.py", line 293, in _wrap_coverage
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 325, in _wrap_profiling
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 410, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File ".bootstrap/_pex/pex.py", line 468, in execute_entry
    return runner(entry_point)
  File ".bootstrap/_pex/pex.py", line 473, in execute_module
    runpy.run_module(module_name, run_name='__main__')
  File "/usr/lib64/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/main.py", line 407, in <module>
    main()
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/main.py", line 402, in main
    add_networks
  File "/root/.pex/install/argh-0.26.2-py2.py3-none-any.whl.b72e563a26b7beab350f4aec4d89dcd75abbee79/argh-0.26.2-py2.py3-none-any.whl/argh/dispatching.py", line 328, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/root/.pex/install/argh-0.26.2-py2.py3-none-any.whl.b72e563a26b7beab350f4aec4d89dcd75abbee79/argh-0.26.2-py2.py3-none-any.whl/argh/dispatching.py", line 174, in dispatch
    for line in lines:
  File "/root/.pex/install/argh-0.26.2-py2.py3-none-any.whl.b72e563a26b7beab350f4aec4d89dcd75abbee79/argh-0.26.2-py2.py3-none-any.whl/argh/dispatching.py", line 277, in _execute_command
    for line in result:
  File "/root/.pex/install/argh-0.26.2-py2.py3-none-any.whl.b72e563a26b7beab350f4aec4d89dcd75abbee79/argh-0.26.2-py2.py3-none-any.whl/argh/dispatching.py", line 260, in _call
    result = function(*positional, **keywords)
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/main.py", line 285, in install
    component.install()
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 186, in install
    self._configure()
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 180, in _configure
    self._start_rabbitmq()
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 158, in _start_rabbitmq
    self._set_policies()
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 134, in _set_policies
    policy=logs_queue_message_policy
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 109, in _set_rabbitmq_policy
    'queues'])
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/components/rabbitmq/rabbitmq.py", line 66, in _rabbitmqctl
    return sudo([RABBITMQ_CTL, '-n', nodename] + command, **kwargs)
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/utils/common.py", line 65, in sudo
    return run(command=command, *args, **kwargs)
  File "/root/.pex/install/cloudify_manager_install-4.5-py2-none-any.whl.28c7f575154a1ede702d9aa2574627cbc42f8593/cloudify_manager_install-4.5-py2-none-any.whl/cfy_manager/utils/common.py", line 54, in run
    raise BootstrapError(msg)
BootstrapError: Failed running command: sudo rabbitmqctl -n cloudify-manager@localhost set_policy logs_queue_message_policy ^cloudify-logs$ {"message-ttl": 60000, "max-length": 1000000} --apply-to queues (Error: unable to perform an operation on node 'cloudify-manager@localhost'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node cloudify-manager@localhost

DIAGNOSTICS
===========

attempted to contact: ['cloudify-manager@localhost']

cloudify-manager@localhost:
  * connected to epmd (port 4369) on localhost
  * node cloudify-manager@localhost up, 'rabbit' application running

Current node details:
 * node name: 'rabbitmqcli17@metaswitch-commissioning-cfy'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: n5fzgvnnW0mW5gApc/ZRRg==

).

The relevant code is https://github.com/cloudify-cosmo/cloudify-manager-install/blob/master/cfy_manager/components/rabbitmq/rabbitmq.py#L123-127

So the stderr is unable to perform an operation on node 'cloudify-manager@localhost'

Seems odd that we are successfully able to run e.g ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'set_user_tags', 'cloudify', 'administrator'] but we fail to run ['sudo', 'rabbitmqctl', '-n', 'cloudify-manager@localhost', 'set_policy', 'logs_queue_message_policy', '^cloudify-logs$', '{"message-ttl": 60000, "max-length": 1000000}', '--apply-to', 'queues']

That suggests that it probably isn't a 'rabbitMQ isn't ready to accept requests' condition.

Running this manually I get:

[admin@host-172-16-0-61 ~]$ sudo rabbitmqctl -n cloudify-manager@localhost set_policy logs_queue_message_policy ^cloudify-logs$ {"message-ttl": 60000, "max-length": 1000000} --apply-to queues
Error (argument validation): too many arguments.
Arguments given:
	-n cloudify-manager@localhost set_policy logs_queue_message_policy ^cloudify-logs$ {message-ttl: 60000, max-length: 1000000} --apply-to queues

Usage:
rabbitmqctl [-n <node>] [-l] [-q] set_policy [-p <vhost>] [--priority <priority>] [--apply-to <apply-to>] <name> <pattern> <definition>

Two problems here:
(1) --apply-to queues is in the wrong place
(2) I worry the ^ $ and {} aren't nicely escaped by shlex / subprocess
Indeed, when I run:

[admin@host-172-16-0-61 ~]$ sudo rabbitmqctl -n cloudify-manager@localhost set_policy --apply-to queues logs_queue_message_policy '^cloudify-logs$' '{"message-ttl": 60000, "max-length": 1000000}'
Setting policy "logs_queue_message_policy" for pattern "^cloudify-logs$" to "{"message-ttl": 60000, "max-length": 1000000}" with priority "0" for vhost "/" ...

This works.

However, that isn't consistent with the error we got about unable to perform an operation on node 'cloudify-manager@localhost'

[mcouthon]

@andrew-lee-metaswitch I suggest contacting Cloudify support directly. I'm not sure GitHub issues are will monitored.