Errors when Indexing, Rebuilding or Restarting

forrest

05 Mar, 2013 05:17 PM

About 8 hours ago I started receiving errors when my application attempted to run rake fs:index. While troubleshooting I attempted to run rake fs:rebuild which appeared to hang. Now my application is also throwing connection refused errors when attempting to query fs.

When running rake fs:index, this is the error returned:
Faraday::Error::ParsingError: 757: unexpected token at '<!DOCTYPE html> We're sorry, but something went wrong (500) body { background-color: #fff; color: #666; text-align: center; font-family: arial, sans-serif; } div.dialog { width: 25em; padding: 0 4em; margin: 4em auto 0 auto; border: 1px solid #ccc; border-right-color: #999; border-bottom-color: #999; } h1 { font-size: 100%; color: #f00; line-height: 1.5em; } <!-- This file lives in public/500.html -->

We're sorry, but something went wrong.

We've been notified about this issue and we'll take a look at it shortly.

'

The error I'm receiving when querying is:
Riddle::ConnectionError: Connection to ec2-174-###-##-##.compute-1.amazonaws.com on 9335 failed. Connection refused - connect(2)

  1. #1 Posted by forrest on 05 Mar, 2013 06:11 PM

    To me this looks very similar to this socket issue: http://support.flying-sphinx.com/discussions/problems/359-flying-sp...

  2. #2 Posted by Pat Allan Support staff on 06 Mar, 2013 01:05 AM

    Hi Forrest

    I'm at a bit of a loss why this is/was happening so consistently for you (though it looks like your daemon is running properly again now). Can you try updating to the latest flying-sphinx and see if that helps?

       gem 'flying-sphinx',
         :git => 'git://github.com/flying-sphinx/flying-sphinx.git',
         :ref => 'b4c81a6d83'

    There'll be a proper gem release soon, I'm just waiting on one of the dependency gems to fix something and get a new release out.

    Kind regards

    Pat

  3. #3 Posted by forrest on 06 Mar, 2013 03:28 PM

    Hi Pat,

    I continued to receive the errors after your post last night. This morning I updated the gem and am now seeing "Action timed out. If this is happening regularly, please contact Flying Sphinx support." when running heroku run rake fs:index.

    If I run heroku run rake fs:restart:
    Executing Action: restart
    Action timed out. If this is happening regularly, please contact Flying Sphinx support: http://support.flying-sphinx.com
    Action Finished: restart

    Take a look and let me know if there's something I can do on my end. I did not initially try updating gems, etc on my end because I had not made any changes to my application over the weekend when I started seeing these errors, so I assumed there had been a change on the flying sphinx side.

    Thanks in advance for your time,

    Forrest

  4. #4 Posted by georgegwu on 06 Mar, 2013 03:39 PM

    I am having exactly the same error running rebuild just now and my production is down as a result.

    Then I also tried

    heroku run rake fs:index --app slm-production --trace

    and still got below error:

    Running rake fs:index --trace attached to terminal... up, run.3020
    ** Invoke fs:index (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute fs:index Starting Index Request
    rake aborted!
    757: unexpected token at '<!DOCTYPE html>


    We're sorry, but something went wrong (500) body { background-color: #fff; color: #666; text-align: center; font-family: arial, sans-serif; } div.dialog { width: 25em; padding: 0 4em; margin: 4em auto 0 auto; border: 1px solid #ccc; border-right-color: #999; border-bottom-color: #999; } h1 { font-size: 100%; color: #f00; line-height: 1.5em; }


    <!-- This file lives in public/500.html -->

    We're sorry, but something went wrong.

    We've been notified about this issue and we'll take a look at it shortly.



    ' /app/vendor/bundle/ruby/1.9.1/gems/json-1.7.6/lib/json/common.rb:155:in parse' /app/vendor/bundle/ruby/1.9.1/gems/json-1.7.6/lib/json/common.rb:155:inparse' /app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.4/lib/multi_json/engines/json_common.rb:9:in decode' /app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.4/lib/multi_json.rb:76:indecode' /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.7.0/lib/faraday/response/parse_json.rb:16:in parse' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/response.rb:17:inon_complete' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/response.rb:9:in block in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/response.rb:62:inon_complete' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/response.rb:8:in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/response.rb:8:incall' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/request/url_encoded.rb:14:in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/connection.rb:207:inrun_request' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.5/lib/faraday/connection.rb:99:in put' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/api.rb:34:inblock in put' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/api.rb:68:in call' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/api.rb:68:inlog' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/api.rb:33:in put' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/index_request.rb:70:inupdate_sphinx_configuration' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/index_request.rb:32:in update_and_index' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.6.1/lib/flying_sphinx/tasks.rb:6:inblock (2 levels) in ' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:228:in call' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:228:inblock in execute' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:223:in each' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:223:inexecute' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:166:in block in invoke_with_call_chain' /usr/local/lib/ruby/1.9.1/monitor.rb:201:inmon_synchronize' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:159:in invoke_with_call_chain' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:152:ininvoke' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:143:in invoke_task' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:inblock (2 levels) in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:in each' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:inblock in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:110:in run_with_threads' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:95:intop_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:73:in block in run' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:160:instandard_exception_handling' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:70:in run' /app/vendor/bundle/ruby/1.9.1/gems/rake-10.0.3/bin/rake:33:in' /app/vendor/bundle/ruby/1.9.1/bin/rake:19:in load' /app/vendor/bundle/ruby/1.9.1/bin/rake:19:in' Tasks: TOP => fs:index

    Please advise. Thanks,

    George

  5. #5 Posted by Pat Allan Support staff on 06 Mar, 2013 04:00 PM

    Hi Forrest, George

    I've been working on this problem the last few hours solidly, and found some clues as to what the cause might be. I'm pretty sure the issue is now resolved, if you could run a rebuild to confirm, that'd be fantastic.

    Kind regards

    Pat

  6. #6 Posted by forrest on 06 Mar, 2013 04:03 PM

    Pat, this fixed it on my end -- I've successfully run fs:index and fs:rebuild commands. Thanks for your extra time on this, especially these late night hours!

  7. #7 Posted by georgegwu on 06 Mar, 2013 04:06 PM

    worked now! Pat> Thanks for the quick response. Really appreciated.

  8. #8 Posted by Pat Allan Support staff on 06 Mar, 2013 04:11 PM

    Great to hear it's now back to normal for both of you. Time for me to get some sleep (03:10 here), but if anything else does crop up, let me know :)

  9. #9 Posted by forrest on 29 Mar, 2013 02:33 PM

    Hi Pat, I started seeing timeouts again this morning, quite inexplicably (once again, I haven't pushed code to production today). I'm posting on this thread as I wonder if it's related to my using the "b4c81a6d83" flying-sphinx gem still.

    Thanks in advance,

    Forrest

  10. #10 Posted by Pat Allan Support staff on 29 Mar, 2013 02:44 PM

    Hi Forrest

    The server your Sphinx setup is hosted on stopped responding about 40 minutes ago. I'm in the process of bringing up a new server with the backups (as well as rebooting the original server, just in case that's faster - though I've never found that to be the case). Hoping to have things back to normal in 20 minutes or so with a bit of luck.

    Will let you know when it's sorted (also posting updates to Twitter at @flying_sphinx).

    Pat

  11. #11 Posted by George Wu on 29 Mar, 2013 02:47 PM

    Pat>

    I am seeing the same error here.

  12. #12 Posted by forrest on 29 Mar, 2013 02:48 PM

    Thanks for the reply here, Pat. Will continue to follow @flying-sphinx and check there in future before posting.

  13. #13 Posted by Pat Allan Support staff on 29 Mar, 2013 02:59 PM

    George, Forrest, the new server is now in place. Is everything running smoothly for your apps again?

  14. #14 Posted by George Wu on 29 Mar, 2013 03:01 PM

    yes. it is now. Thanks.

  15. #15 Posted by forrest on 29 Mar, 2013 03:04 PM

    yes, everything is running fine again.

  16. #16 Posted by Pat Allan Support staff on 29 Mar, 2013 03:10 PM

    Great to hear. I'll keep a close eye on things, though I'm not sure what was the cause of the problem - server had been running for nine months without downtime, and New Relic didn't alert me to high CPU/IO activity (and they're normally pretty good at that).

    Still, backups (which are updated for each app when a full index is run) were brought up without issue, it was just a matter of waiting for them to copy over and all dependencies to be installed on the server.

    Appreciate your understanding and patience with this.

    Kind regards,

    Pat

  17. forrest closed this discussion on 18 Jul, 2014 03:49 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles