"flying-sphinx start" results in Faraday::Error::ParsingError (for internal server error response)

I'm trying to start a Flying-Sphinx (Wooden) but getting Faraday::Error::ParsingError

when I run heroku run flying-sphinx start it produces:

Running `flying-sphinx start` attached to terminal... up, run.1
[2012-10-01 16:13:59 UTC] API Request: GET '/'; params: {}
[2012-10-01 16:13:59 UTC] API Response: #<Hashie::Rash database_port=10318 port=9318 server="ec2-23-22-176-191.compute-1.amazonaws.com" ssh_server="ec2-23-22-176-191.compute-1.amazonaws.com" status="OK">
[2012-10-01 16:13:59 UTC] API Request: POST 'start'; params: {}
/app/vendor/bundle/ruby/1.9.1/gems/json-1.7.5/lib/json/common.rb:155:in `parse': 757: unexpected token at '<!DOCTYPE html> (Faraday::Error::ParsingError)
<html>
<head>
  <title>We're sorry, but something went wrong (500)</title>
  <style type="text/css">
    body { background-color: #fff; color: #666; text-align: center; font-family: arial, sans-serif; }
    div.dialog {
      width: 25em;
      padding: 0 4em;
      margin: 4em auto 0 auto;
      border: 1px solid #ccc;
      border-right-color: #999;
      border-bottom-color: #999;
    }
    h1 { font-size: 100%; color: #f00; line-height: 1.5em; }
  </style>
</head>

<body>
  <!-- This file lives in public/500.html -->
  <div class="dialog">
    <h1>We're sorry, but something went wrong.</h1>
    <p>We've been notified about this issue and we'll take a look at it shortly.</p>
  </div>
</body>
</html>
'
    from /app/vendor/bundle/ruby/1.9.1/gems/json-1.7.5/lib/json/common.rb:155:in `parse'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response/parse_json.rb:11:in `block in <class:ParseJson>'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response_middleware.rb:48:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response_middleware.rb:48:in `parse'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response_middleware.rb:39:in `process_response'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response_middleware.rb:32:in `block in call'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.8.4/lib/faraday/response.rb:63:in `on_complete'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.8.8/lib/faraday_middleware/response_middleware.rb:30:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.8.4/lib/faraday/response.rb:8:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.8.4/lib/faraday/request/url_encoded.rb:14:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.8.4/lib/faraday/connection.rb:226:in `run_request'
    from /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.8.4/lib/faraday/connection.rb:99:in `post'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/api.rb:23:in `block in post'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/api.rb:69:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/api.rb:69:in `log'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/api.rb:22:in `post'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/configuration.rb:16:in `start_sphinx'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/cli.rb:64:in `start'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/cli.rb:20:in `block in run'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/cli.rb:20:in `each'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/cli.rb:20:in `all?'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/lib/flying_sphinx/cli.rb:20:in `run'
    from /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.8.4/bin/flying-sphinx:5:in `<top (required)>'
    from bin/flying-sphinx:16:in `load'
    from bin/flying-sphinx:16:in `<main>'
  1. This is certainly not expected...

    What's your app name? Also, what versions of Thinking Sphinx and Rails are you using?

    Also, can you try updating flying-sphinx to use this commit from git://github.com/flying-sphinx/flying-sphinx.git
    6e93c5f6a94bbbf0ad1bd0251273f9ce0f319094

    Apologies for the delay and brevity, currently on a farm in Wales for a few days.

    --
    Pat

  2. hi,

    I've been having the same issue since this morning (all of a sudden, didnt change a thing). We have two heroku servers one for staging and one for production. Same issue on both.

    I tried the latest rep from: https://github.com/flying-sphinx/flying-sphinx/commits/master in vain. Kindly help me out. If you need more info, kindly let me know.

    Regards,
    Hassan

  3. Hi Hassan

    Still investigating to find the cause, but there was something causing a significant backlog with job processing (which all actions go through - and normally there's no backlog at all!). If indexing is still timing out or slow, do let me know, because I'd like to get this sorted.

    Names for the apps that are seeing issues would be greatly appreciated.

    I'll continue to hunt through logs and New Relic data to see if I can figure out the original cause though.

    Kind regards,

    Pat Allan

  4. Hi Hassan

    Just further to this: It's a bit hard to tell what sparked this problem - New Relic and logging provide some clues, but here's what I think happened:

    • Someone signed up for Flying Sphinx and then created an enormous amount of index requests (not great for the queue).
    • They then removed their account - which meant the index requests kept looking for the account, failed, and were retrying (also not great for the queue).

    This hurt the service for everyone else, so here's what's changing:

    • If accounts are removed, index requests (and other actions) will fail gracefully and not retry (already in production!).
    • I'm considering implementing rate limiting for indexing… adds a bit more overhead though, but maybe there's a way that it can happen outside the API HTTP requests, to keep them snappy.

    But again: if you're still having issues (or anything else crops up in the future), do let me know.

    I appreciate your patience!

Reply to this discussion Internal reply only staff will see it

Markdown

Attaching KB article:

» »

Read more…

Attach files

Up to 10MB each.

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions