Connection failure

Matt Olson

04 May, 2015 03:50 AM

Hi,

I'm setting up a new instance with Thinking Sphinx 3.1.3 and Flying Sphinx 1.2.0, and getting the following error:

2015-05-04T03:44:47.695026+00:00 app[web.1]: ThinkingSphinx::ConnectionError (Error connecting to Sphinx via the MySQL protocol. Error connecting to Sphinx via the MySQL protocol. Can't connect to MySQL server on '127.0.0.1' (111) - SELECT * FROM `title_core` WHERE MATCH('@full_title harr') AND `sphinx_deleted` = 0 ORDER BY `full_title` ASC LIMIT 0, 20; SHOW META):

Shouldn't the Flying Sphinx gem pick up the connection settings from environment variables? Not sure what's going on here. I was able to run heroku run bundle exec flying-sphinx configure and heroku run bundle exec flying-sphinx rebuild without any problems.

Help?

The app name is glacial-mesa-4031 if that helps.

Thanks,
Matt

  1. #1 Posted by Pat Allan Support staff on 04 May, 2015 04:20 AM

    Hi Matt

    The Flying Sphinx gem should indeed pick up the environment variables. I'm a little surprised why it currently isn't doing that for you. I presume the environment vars are set? (There should be four - api key, host, identifier, port.)

    And I presume restarting the app hasn't helped fix this either?

    Kind regards,

    Pat

  2. #2 Posted by Matt Olson on 04 May, 2015 04:28 AM

    Yes, FLYING_SPHINX_API_KEY, FLYING_SPHINX_HOST, FLYING_SPHINX_IDENTIFIER, and FLYING_SPHINX_PORT are all set. And yes, I've tried restarting the app.

    It's puzzling that I can run flying-sphinx commands successfully (which presumably read from these same environment variables), but when I run the app, Thinking Sphinx tries to connect to 127.0.0.1.

    Any ideas?

  3. #3 Posted by Pat Allan Support staff on 04 May, 2015 04:31 AM

    Would it be possible for you to add me to the app to debug directly? My Heroku account is [email blocked].

    If this is not an option, that's completely understandable... I'll keep pondering either way.

  4. #4 Posted by Matt Olson on 04 May, 2015 04:48 AM

    Sorry, Pat, but I don't think I can do that.

    However, here's another mystery: I just restarted the app after switching from Github master for Flying Sphinx to the 1.2.0 gem, and got this instead:

    2015-05-04T04:46:11.001853+00:00 app[web.1]: ThinkingSphinx::SphinxError (Lost connection to MySQL server at 'reading authorization packet', system error: 0 - SELECT * FROM `title_core` WHERE MATCH('@full_title hayy') AND `sphinx_deleted` = 0 ORDER BY `full_title` ASC LIMIT 0, 20; SHOW META):

  5. #5 Posted by Pat Allan Support staff on 04 May, 2015 04:50 AM

    Definitely best to stick to the 1.2.0 release for the moment, instead of the latest in master. That error is perhaps a little more promising - though it indicates Sphinx isn't running on the Flying Sphinx side of things (and from what I can see, it is). Will dive deeper on the servers to see what might be going wrong.

  6. #6 Posted by Matt Olson on 04 May, 2015 04:53 AM

    OK, good to know. I think things are working now! Did you do something on your side? This sequence of events is mysterious to me...

  7. #7 Posted by Pat Allan Support staff on 04 May, 2015 04:57 AM

    I hadn't even got to SSH'ing into the server! But here's my thinking:

    • There must be a bug in master since the 1.2.0 release of the flying-sphinx gem.
    • The very first connection hits the Sphinx proxy on the appropriate Flying Sphinx server, which hasn't seem these credentials before. It drops that connection, but also asynchronously updates expected credentials.
    • Hence, further search requests are working, because the credentials are now cached.

    Great to have it working now... I'll see if I can fix whatever the bug in recent commits is.

  8. #8 Posted by Matt Olson on 04 May, 2015 05:13 AM

    OK, sounds like a plan.

    On another note, while I have your attention... :)

    I'm planning to upgrade another instance from Flying Sphinx 0.8.4 to 1.2.0 and Thinking Sphinx 2.0.13 to 3.1.3. (The instance in the original post is a new staging server to test the changes.) Apart from flying-sphinx rebuild, are there any other steps I should take when pushing this change? I've already made any code changes, such as moving the index definition from the model to app/indices. Everything seems to be working now on staging, just wondering if there are any gotchas from the Flying Sphinx side of things if I upgrade the gems.

    Thanks,
    Matt

  9. #9 Posted by Pat Allan Support staff on 04 May, 2015 05:46 AM

    Hi Matt

    I presume you’ve got config/thinking_sphinx.yml using a modern version of Sphinx? 2.1.x is certainly recommended - available versions are listed here: http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version <http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version>

    I’d expect the flow to be:
    * deploy code that include the gem updates
    * run Sphinx rebuild
    * Maybe see a single error message post-rebuild, like previously?
    * All fine from that point onwards.

    I wouldn’t expect anything to go particularly pear-shaped, but happy to be on hand if it’s your evening when you roll it all out.

    Cheers

    —
    Pat

  10. #10 Posted by Matt Olson on 07 May, 2015 05:00 AM

    Hey Pat,

    FYI, I just upgraded TS and FS without a hitch, just did a rebuild after deploy and everything went smoothly.

    Thanks,
    Matt

  11. #11 Posted by Pat Allan Support staff on 07 May, 2015 05:02 AM

    Great to hear :)

  12. #12 Posted by Pat Allan Support staff on 08 May, 2015 11:11 AM

    Hi Matt

    Just spending some time trying to reproduce your initial issue, but no luck yet. Which version of Rails are you using?

    Thanks

    Pat

  13. #13 Posted by Matt Olson on 08 May, 2015 02:31 PM

    I was using 3.2.16

  14. #14 Posted by Pat Allan Support staff on 09 May, 2015 02:07 AM

    I was on 3.2.18, switched down to 3.2.16, still no issues. My Gemfile in this test app is pretty simple: https://gist.github.com/pat/081500cf3cd6c30f8140

    I'm at a loss as to what's causing the problem... possibly another gem in your Gemfile was getting in the way? Though I really don't get how that could be the case (and I don't think there have been any changes in the way FS works with TS since 1.2.0 was released anyway). At this point, I'm considering this issue as unreproduceable - which is frustrating, but hopefully if it does crop up for anyone else, I'll hear about it and can see if the extra data points help.

    Appreciate your patience with all of this Matt - and at least things have been working smoothly for you since switching to the 1.2.0 release.

  15. Pat Allan closed this discussion on 09 May, 2015 02:07 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles