Can't index or rebuild after a db:reset

Corbin's Avatar

Corbin

30 May, 2013 04:04 PM

I'm using flying sphinx to use thinking-sphinx on heroku.

It's been working great, but I was having database problems and I ran a rake db:reset to get a clean slate.

Since I did this, flying sphinx isn't indexing or rebuilding. This problem is only happening on my staged site, the thinking-sphinx works fine in my local dev environment.

After I type:

>> heroku run bundle exec flying-sphinx index

I get..

Running `bundle exec flying-sphinx index` attached to terminal... up, run.9005
I, [2013-05-30T15:56:45.137261 #2] INFO -- : Executing Action: index

Then, it freezes and doesn't index. Same with rebuilding.

What do you think may be the problem?

  1. 1 Posted by Corbin on 30 May, 2013 04:28 PM

    Corbin's Avatar

    Update:

    It looks like db:reset doesn't work on heroku, because of permissions.

    Tried again with pg:reset and db:setup, and the problem still exists.

  2. 2 Posted by Corbin on 30 May, 2013 04:43 PM

    Corbin's Avatar

    just ran
    >> heroku run bundle exec flying-sphinx configure

    And got the output...

    Connecting to database specified by DATABASE_URL
    W, [2013-05-30T16:41:42.166902 #2] WARN -- : Action timed out. If this is happening regularly, please contact Flying Sphinx support: http://support.flying-sphinx.com
    I, [2013-05-30T16:41:42.167176 #2] INFO -- : Action Finished: configure

  3. 3 Posted by Sandi Lin on 30 May, 2013 04:51 PM

    Sandi Lin's Avatar

    I have also been unable to index flying-sphinx on Heroku for the last 12 hours. Same problem of freezing. I'm not sure whether it's related, but maybe there's a more systematic problem going on?

  4. 4 Posted by Corbin on 30 May, 2013 04:56 PM

    Corbin's Avatar

    @Sandi Thanks for the comment. I'll stop beating my head against the wall until I hear from Pat on this. :)

    I also reinstalled flying sphinx, with no luck. I'm out of ideas...

  5. 5 Posted by Sandi Lin on 30 May, 2013 06:29 PM

    Sandi Lin's Avatar

    I'm getting as far as rotating indices now:
    rotating indices: successfully sent SIGHUP to searchd

    But still not completing.

  6. 6 Posted by Corbin on 30 May, 2013 09:00 PM

    Corbin's Avatar

    Magic! It's working now.

    @FlyingSphinx why did this happen?

    BTW I'm running wooden on vienna

  7. 7 Posted by Sandi Lin on 30 May, 2013 09:07 PM

    Sandi Lin's Avatar

    That's good to hear. Mine is still freezing. I'm also using wooden.

  8. 8 Posted by Sandi Lin on 30 May, 2013 09:17 PM

    Sandi Lin's Avatar

    Correction: seems to be working but is very slow and occasionally still times out.

  9. Support Staff 9 Posted by Pat Allan on 30 May, 2013 11:42 PM

    Pat Allan's Avatar

    Hi Sandi, Corbin

    Still investigating to find the cause, but there was something causing a significant backlog with job processing (which all actions go through - and normally there's no backlog at all!). If indexing is still timing out or slow, do let me know, because I'd like to get this sorted.

    I'll continue to hunt through logs and New Relic data to see if I can figure out the original cause though.

    Kind regards,

    Pat Allan

  10. 10 Posted by Sandi Lin on 31 May, 2013 12:16 AM

    Sandi Lin's Avatar

    Thanks Pat for looking into it! Good luck!

  11. Support Staff 11 Posted by Pat Allan on 31 May, 2013 12:40 AM

    Pat Allan's Avatar

    It's a bit hard to tell - New Relic and logging provide some clues, but here's what I think happened:

    * Someone signed up for Flying Sphinx and then created an enormous amount of index requests (not great for the queue).
    * They then removed their account - which meant the index requests kept looking for the account, failed, and were retrying (also not great for the queue).

    This hurt the service for everyone else, so here's what's changing:

    * If accounts are removed, index requests (and other actions) will fail gracefully and not retry (already in production!).
    * I'm considering implementing rate limiting for indexing… adds a bit more overhead though, but maybe there's a way that it can happen outside the API HTTP requests, to keep them snappy.

    But again: if you're still having issues (or anything else crops up in the future), do let me know.

    I appreciate your patience!

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac