I'm using flying sphinx to use thinking-sphinx on heroku.
It's been working great, but I was having database problems and I ran a rake db:reset to get a clean slate.
Since I did this, flying sphinx isn't indexing or rebuilding. This problem is only happening on my staged site, the thinking-sphinx works fine in my local dev environment.
After I type:
>> heroku run bundle exec flying-sphinx index
I get..
Running `bundle exec flying-sphinx index` attached to terminal... up, run.9005
I, [2013-05-30T15:56:45.137261 #2] INFO -- : Executing Action: index
Then, it freezes and doesn't index. Same with rebuilding.
What do you think may be the problem?
#1 Posted by Corbin on 30 May, 2013 04:28 PM
Update:
It looks like db:reset doesn't work on heroku, because of permissions.
Tried again with pg:reset and db:setup, and the problem still exists.
#2 Posted by Corbin on 30 May, 2013 04:43 PM
just ran
>> heroku run bundle exec flying-sphinx configure
And got the output...
Connecting to database specified by DATABASE_URL
W, [2013-05-30T16:41:42.166902 #2] WARN -- : Action timed out. If this is happening regularly, please contact Flying Sphinx support: http://support.flying-sphinx.com
I, [2013-05-30T16:41:42.167176 #2] INFO -- : Action Finished: configure
#3 Posted by Sandi Lin on 30 May, 2013 04:51 PM
I have also been unable to index flying-sphinx on Heroku for the last 12 hours. Same problem of freezing. I'm not sure whether it's related, but maybe there's a more systematic problem going on?
#4 Posted by Corbin on 30 May, 2013 04:56 PM
@Sandi Thanks for the comment. I'll stop beating my head against the wall until I hear from Pat on this. :)
I also reinstalled flying sphinx, with no luck. I'm out of ideas...
#5 Posted by Sandi Lin on 30 May, 2013 06:29 PM
I'm getting as far as rotating indices now:
rotating indices: successfully sent SIGHUP to searchd
But still not completing.
#6 Posted by Corbin on 30 May, 2013 09:00 PM
Magic! It's working now.
@FlyingSphinx why did this happen?
BTW I'm running wooden on vienna
#7 Posted by Sandi Lin on 30 May, 2013 09:07 PM
That's good to hear. Mine is still freezing. I'm also using wooden.
#8 Posted by Sandi Lin on 30 May, 2013 09:17 PM
Correction: seems to be working but is very slow and occasionally still times out.
#9 Posted by Pat Allan Support staff on 30 May, 2013 11:42 PM
Hi Sandi, Corbin
Still investigating to find the cause, but there was something causing a significant backlog with job processing (which all actions go through - and normally there's no backlog at all!). If indexing is still timing out or slow, do let me know, because I'd like to get this sorted.
I'll continue to hunt through logs and New Relic data to see if I can figure out the original cause though.
Kind regards,
Pat Allan
#10 Posted by Sandi Lin on 31 May, 2013 12:16 AM
Thanks Pat for looking into it! Good luck!
#11 Posted by Pat Allan Support staff on 31 May, 2013 12:40 AM
It's a bit hard to tell - New Relic and logging provide some clues, but here's what I think happened:
* Someone signed up for Flying Sphinx and then created an enormous amount of index requests (not great for the queue).
* They then removed their account - which meant the index requests kept looking for the account, failed, and were retrying (also not great for the queue).
This hurt the service for everyone else, so here's what's changing:
* If accounts are removed, index requests (and other actions) will fail gracefully and not retry (already in production!).
* I'm considering implementing rate limiting for indexing… adds a bit more overhead though, but maybe there's a way that it can happen outside the API HTTP requests, to keep them snappy.
But again: if you're still having issues (or anything else crops up in the future), do let me know.
I appreciate your patience!