Hi, on my fh7020 heroku app sphinx stopped working suddenly today.
I keep getting the following error: ThinkingSphinx::SphinxError (Lost connection to MySQL server at 'reading authorization packet', system error: 0):
I tried restart, index, stop/start, nothing helps.
Can you please help? This is our production environment.
-Erhan Berber
610.662.3300
#1 Posted by forrest on 29 Feb, 2016 07:34 PM
We're seeing this as well.
#2 Posted by forrest on 29 Feb, 2016 07:58 PM
To clarify, I'm seeing this in Demo and Staging environments, but not yet in production. Servers appear to respond to configure, rebuild, restart, stop, and start, but no matter, when our application tries to query sphinx, we get this error.
I also tried removing the flying-sphinx add-on from our demo application (hoping that in re-adding it would spin up on a server without the issue) and now when I try to re-add heroku says "Could not communicate with vendor"
#3 Posted by forrest on 29 Feb, 2016 09:30 PM
Still seeing these issues after your update. I was able to re-add sphinx to my Demo instance and configure, index, and start, but as soon as I try to load app pages I'm still seeing the lost connection error.
#4 Posted by Pat Allan Support staff on 29 Feb, 2016 09:34 PM
Erhan, checking on your app now. Forrest: can you let me know the app name please? I'll hunt down the cause of your problem too.
#5 Posted by forrest on 29 Feb, 2016 09:35 PM
Our staging environment, on Averna, is back up now. Still seeing issues with our Demo environment, hosted on Strenua.
Our demo app is lexspot2-demo
#6 Posted by Erhan Berber on 29 Feb, 2016 09:43 PM
Thanks so much, Pat.
Please let me know, again this is production...
-Erhan
#7 Posted by forrest on 29 Feb, 2016 09:45 PM
Looks like our Demo app is working now. I'll open a new thread if any errors come in going forward. Thanks for your help, Pat.
#8 Posted by Erhan Berber on 29 Feb, 2016 09:49 PM
Our production is working right now, Pat.
Thanks so much for your help.
Could you tell what caused it?
-Erhan
#9 Posted by Pat Allan Support staff on 29 Feb, 2016 09:54 PM
Hi Erhan, Forrest
Good to know things are working again (certainly, my tests just now indicated everything was fine). I'll be looking into the root cause of the issue to find out exactly what's wrong, but at this stage, I believe the database problem at the API caused problems for the Sphinx proxy on each server (which expects authentication for each app) and it couldn't pair your apps to the right daemons because of that API 500 error.
It may have been something else - there'll be an update to this ticket when I'm more certain: http://status.flying-sphinx.com/incidents/6g18cc3cgcnl
I'll also be looking into better alerting for these kinds of database errors so I'll then be woken up by Pingdom immediately, rather than letting the issue grow (it's almost 9am here in Melbourne).
Very sorry for the problems, and greatly appreciate your patience.
Pat
#10 Posted by Erhan Berber on 29 Feb, 2016 11:18 PM
Thanks so much, Pat.
I am here if you may need any help/info from me.
-Erhan