Sphinx stopped working on my site

Erhan Berber's Avatar

Erhan Berber

29 Feb, 2016 07:26 PM

Hi, on my fh7020 heroku app sphinx stopped working suddenly today.

I keep getting the following error: ThinkingSphinx::SphinxError (Lost connection to MySQL server at 'reading authorization packet', system error: 0):

I tried restart, index, stop/start, nothing helps.

Can you please help? This is our production environment.

-Erhan Berber

  1. 1 Posted by forrest on 29 Feb, 2016 07:34 PM

    forrest's Avatar

    We're seeing this as well.

  2. 2 Posted by forrest on 29 Feb, 2016 07:58 PM

    forrest's Avatar

    To clarify, I'm seeing this in Demo and Staging environments, but not yet in production. Servers appear to respond to configure, rebuild, restart, stop, and start, but no matter, when our application tries to query sphinx, we get this error.

    I also tried removing the flying-sphinx add-on from our demo application (hoping that in re-adding it would spin up on a server without the issue) and now when I try to re-add heroku says "Could not communicate with vendor"

  3. 3 Posted by forrest on 29 Feb, 2016 09:30 PM

    forrest's Avatar

    Still seeing these issues after your update. I was able to re-add sphinx to my Demo instance and configure, index, and start, but as soon as I try to load app pages I'm still seeing the lost connection error.

  4. Support Staff 4 Posted by Pat Allan on 29 Feb, 2016 09:34 PM

    Pat Allan's Avatar

    Erhan, checking on your app now. Forrest: can you let me know the app name please? I'll hunt down the cause of your problem too.

  5. 5 Posted by forrest on 29 Feb, 2016 09:35 PM

    forrest's Avatar

    Our staging environment, on Averna, is back up now. Still seeing issues with our Demo environment, hosted on Strenua.

    Our demo app is lexspot2-demo

  6. 6 Posted by Erhan Berber on 29 Feb, 2016 09:43 PM

    Erhan Berber's Avatar

    Thanks so much, Pat.

    Please let me know, again this is production...


  7. 7 Posted by forrest on 29 Feb, 2016 09:45 PM

    forrest's Avatar

    Looks like our Demo app is working now. I'll open a new thread if any errors come in going forward. Thanks for your help, Pat.

  8. 8 Posted by Erhan Berber on 29 Feb, 2016 09:49 PM

    Erhan Berber's Avatar

    Our production is working right now, Pat.

    Thanks so much for your help.

    Could you tell what caused it?


  9. Support Staff 9 Posted by Pat Allan on 29 Feb, 2016 09:54 PM

    Pat Allan's Avatar

    Hi Erhan, Forrest

    Good to know things are working again (certainly, my tests just now indicated everything was fine). I'll be looking into the root cause of the issue to find out exactly what's wrong, but at this stage, I believe the database problem at the API caused problems for the Sphinx proxy on each server (which expects authentication for each app) and it couldn't pair your apps to the right daemons because of that API 500 error.

    It may have been something else - there'll be an update to this ticket when I'm more certain: http://status.flying-sphinx.com/incidents/6g18cc3cgcnl

    I'll also be looking into better alerting for these kinds of database errors so I'll then be woken up by Pingdom immediately, rather than letting the issue grow (it's almost 9am here in Melbourne).

    Very sorry for the problems, and greatly appreciate your patience.


  10. 10 Posted by Erhan Berber on 29 Feb, 2016 11:18 PM

    Erhan Berber's Avatar

    Thanks so much, Pat.

    I am here if you may need any help/info from me.


Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac