Celery + RabbitMQ stuck at mingle: searching for neighbors -
i encountered strange issue while trying system running on new machines.
os:
ubuntu 14.04.1 lts (gnu/linux 3.13.0-39-generic x86_64)
packages:
celery==3.1.13
django-celery==3.0.23
librabbitmq==1.5.2
broker:
rabbitmq "3.2.4"
after restart celery process gets stuck:
[2014-11-10 18:32:55,792: info/mainprocess] connected amqp://user:**@172.16.10.6:5672/vhost [2014-11-10 18:32:55,804: info/mainprocess] mingle: searching neighbors
tried find solution elsewhere. here found advice increase disk space available rabbitmq. checked docs , changed in rabbitmq.config file:
{disk_free_limit, {mem_relative, 1.0}}
now rabbitmq should have 6gb available, doesn't push celery ready state. tried configuring (as suggested) limit of open file descriptors in
/etc/default/rabbitmq-server
without effect. current rabbitmq status:
[{pid,4131}, {running_applications, [{rabbitmq_management,"rabbitmq management console","3.2.4"}, {rabbitmq_web_dispatch,"rabbitmq web dispatcher","3.2.4"}, {webmachine,"webmachine","1.10.3-rmq3.2.4-gite9359c7"}, {mochiweb,"mochimedia web server","2.7.0-rmq3.2.4-git680dba8"}, {rabbitmq_management_agent,"rabbitmq management agent","3.2.4"}, {rabbit,"rabbitmq","3.2.4"}, {os_mon,"cpo cxc 138 46","2.2.14"}, {inets,"inets cxc 138 49","5.9.7"}, {mnesia,"mnesia cxc 138 12","4.11"}, {amqp_client,"rabbitmq amqp client","3.2.4"}, {xmerl,"xml parser","1.3.5"}, {sasl,"sasl cxc 138 11","2.3.4"}, {stdlib,"erts cxc 138 10","1.19.4"}, {kernel,"erts cxc 138 10","2.16.4"}]}, {os,{unix,linux}}, {erlang_version, "erlang r16b03 (erts-5.10.4) [source] [64-bit] [smp:4:4] [async-threads:30] [kernel-poll:true]\n"}, {memory, [{total,46342000}, {connection_procs,290304}, {queue_procs,62864}, {plugins,458136}, {other_proc,13673776}, {mnesia,76376}, {mgmt_db,127808}, {msg_index,34384}, {other_ets,1100432}, {binary,5282744}, {code,19974306}, {atom,703377}, {other_system,4557493}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,2503614464}, {disk_free_limit,6259036160}, {disk_free,2779140096}, {file_descriptors, [{total_limit,924},{total_used,13},{sockets_limit,829},{sockets_used,9}]}, {processes,[{limit,1048576},{used,270}]}, {run_queue,0}, {uptime,4561}]
another thing: when stop celery processes supervisor no warm shutdown logged (actually - nothing logged).
could help?
i had same problem seems. check own answer own question here:
celeryd rabbitmq hangs on "mingle: searching neighbors", plain celery works
long story short:
sudo apt-get remove librabbitmq1
worked me.
Comments
Post a Comment