daemon log showing numerous SIGTERMs

Matt Olson

18 Dec, 2012 08:55 PM

I have been using Wooden from Heroku without trouble for close to a year now, but in the last week or so I have had about a dozen exceptions due to a failure to connect to Sphinx. I am on the "birmingham" server. I just logged in to the Dashboard to look at the daemon log, and have found that the daemon is restarting once a minute, for no apparent reason. Can you help figure out why?

Thanks,
Matt

  1. #1 Posted by Pat Allan Support staff on 18 Dec, 2012 11:00 PM

    Hi Matt

    What's your app's name?

    I know this had been a problem earlier this week for some of the older customers, but I'd pushed out a fix that I thought had resolved it. Perhaps not…

    Pat

  2. #2 Posted by Matt Olson on 18 Dec, 2012 11:04 PM

    stark-cloud-2613

    According to the logs, it was restarting over and over from at least Dec 11 up until 10am this morning. Doesn't appear to be happening right now, but can you double check and offer an explanation?

    Thanks,
    Matt

  3. #3 Posted by Pat Allan Support staff on 18 Dec, 2012 11:20 PM

    Hi Matt

    From what I can see, it hasn't restarted since Monday 01:34 GMT - which is when I pushed out the fix.

    As for why it was happening - I'd made some changes to the Flying Sphinx servers to have a monit daemon monitoring all Sphinx daemons. As part of the configuration, I included a connection test at 127.0.0.1 on the appropriate port. However, people using old versions of the flying-sphinx gem (0.6.x or older) have their app configured through an old version of the flying-sphinx.com API, which instead of configuring Sphinx daemons to bind to 0.0.0.0, binds them to their server's DNS address instead.

    In your case, monit couldn't connect to 127.0.0.1 on 9368, but only ec2-174-129-99-83.compute-1.amazonaws.com on 9368 - and so, it decided the daemon wasn't running and rebooted it.

    The simple fix was: parse the configuration file and have monit use the host the daemon is bound to - which is what I put into play on Monday.

    All that said, you shouldn't have been seeing any connection issues in the past 45 or so hours. If you are, then further investigation is needed.

    Cheers

    Pat

  4. #4 Posted by Matt Olson on 18 Dec, 2012 11:22 PM

    Thanks for the detailed explanation, I appreciate it!

  5. Pat Allan closed this discussion on 13 Jan, 2014 11:24 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles