>> heroku run bundle exec flying-sphinx configure
And got the output...
Connecting to database specified by DATABASE_URL
W, [2013-05-30T16:41:42.166902 #2] WARN -- : Action timed out. If this is happening regularly, please contact Flying Sphinx support: http://support.flying-sphinx.com I, [2013-05-30T16:41:42.167176 #2] INFO -- : Action Finished: configure
Pat Allan on 30 May, 2013 11:42 PM
Hi Sandi, Corbin
Still investigating to find the cause, but there was something causing a significant backlog with job processing (which all actions go through - and normally there's no backlog at all!). If indexing is still timing out or slow, do let me know, because I'd like to get this sorted.
I'll continue to hunt through logs and New Relic data to see if I can figure out the original cause though.
Pat Allan on 31 May, 2013 12:40 AM
It's a bit hard to tell - New Relic and logging provide some clues, but here's what I think happened:
* Someone signed up for Flying Sphinx and then created an enormous amount of index requests (not great for the queue).
* They then removed their account - which meant the index requests kept looking for the account, failed, and were retrying (also not great for the queue).
This hurt the service for everyone else, so here's what's changing:
* If accounts are removed, index requests (and other actions) will fail gracefully and not retry (already in production!).
* I'm considering implementing rate limiting for indexing… adds a bit more overhead though, but maybe there's a way that it can happen outside the API HTTP requests, to keep them snappy.
But again: if you're still having issues (or anything else crops up in the future), do let me know.