Just upgraded to ceramic on heroku. Received migration email on successful migration, but server isn't up and I cannot get it to start.
Seeing errors like this:
[Sun Jun 30 20:41:24.473 2013] [30902] using config file '/mnt/local/flying-sphinx/f3bc8bddd92106b4b/sphinx.conf'...
[Sun Jun 30 20:41:24.474 2013] [30902] WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
listening on all interfaces, port=9328
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
bind() failed on 0.0.0.0, retrying...
FATAL: bind() failed on 0.0.0.0: Address already in use
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
I, [2013-06-30T20:42:03.363183 #2] INFO -- : Action Finished: rebuild
** [NewRelic][06/30/13 20:42:03 +0000 8fada61f-1df9-46a6-a0af-6179402799fc (2)] INFO : Starting Agent shutdown
This seems to indicate to me that the prior version of sphinx wasn't stopped correctly and the new version can't bind with the port. Will continue googling on this to see if I hit on something but any help would be appreciated.
I've tried stopping, starting and rebuilding commands to no avail.
#1 Posted by forrest on 30 Jun, 2013 09:07 PM
Full rebuild action looks like this:
Feste:LexSpot1 forart$ heroku run bundle exec flying-sphinx start -a lexspot2
Running `bundle exec flying-sphinx start` attached to terminal... up, run.6776
I, [2013-06-30T21:06:11.773947 #2] INFO -- : Executing Action: start
[Sun Jun 30 21:06:13.569 2013] [11323] using config file '/mnt/local/flying-sphinx/f3bc8bddd92106b4b/sphinx.conf'...
[Sun Jun 30 21:06:13.570 2013] [11323] WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
FATAL: failed to lock pid file '/mnt/local/flying-sphinx/pids/f3bc8bddd92106b4b.pid': Resource temporarily unavailable (searchd already running?)
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
Sphinx 2.0.6-id64-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)
I, [2013-06-30T21:06:16.114336 #2] INFO -- : Action Finished: start
#2 Posted by forrest on 30 Jun, 2013 09:57 PM
I downgraded back to wooden and after rebuilding everything is working again. I then tried upgrading again, this time explicitly stopping the wooden plan before upgrading, but I ran into the same issues around pid and port, so I downgraded again.
Going to hold off on this upgrade for today -- let me know if you need me to try the upgrade in order to diagnose the issue and we can try to find some time to work on this together.
#3 Posted by Pat Allan Support staff on 01 Jul, 2013 11:13 AM
Hi Forrest
A daemon from a customer who is no longer with Flying Sphinx wasn't killed off properly - a rare occurrence, but certainly frustrating. I've put something in place to help me track down these situations more reliably, and all servers are now free of those rogue searchd daemons - so, nothing's using the port your daemon will use.
Any further issues, do let me know.
Kind regards,
--
Pat
#4 Posted by forrest on 01 Jul, 2013 01:21 PM
Upgraded successfully and seamlessly today. Thanks for fixing this, Pat. Can you clarify for future occurrences, was there an action I could have taken to rectify this process or is it best to rollback immediately if you encounter pid/port issues and wait for you to engage?
#5 Posted by Pat Allan Support staff on 01 Jul, 2013 01:40 PM
Rollback's the only option at this point I'm afraid - but it's a pretty rare occurrence (only found a handful of rogue daemons across a dozen servers). I'll look into better monitoring/cleaning up - perhaps opting for `kill -9` from the beginning, instead of trying less, uh, violent approaches to stopping daemons initially.
#6 Posted by forrest on 01 Jul, 2013 01:44 PM
Better documentation/error messaging would have greatly helped as well -- I didn't find strong hits for the pid or port messaging when searching yesterday so I wasn't confident that the issue wasn't due to a bad config on my end.
Thanks again!
Pat Allan closed this discussion on 13 Jan, 2014 11:23 PM.