searchd error (status: 1): invalid client key

Alex Serdyuk's Avatar

Alex Serdyuk

17 Jul, 2012 06:42 AM

Hi,

I have the following error when trying to use searchd.

FlyingSphinx::FlagAsDeletedJob failed with Riddle::ResponseError: searchd error (status: 1): invalid client key '209a2e7a264af3393:9167e7e5fba407f71' - 2 failed attempts

The error started happening when I swtiched from wooden to ceramic in order to use deltas. The version of flying-sphinx I'm using is the newest 0.7. Any help would be appreciated.

Also, It seems like it stops and indexes fine, but when I start up I get an error (stack trace with --trace):

/app/vendor/bundle/ruby/1.9.1/gems/json-1.7.3/lib/json/common.rb:155:in parse' /app/vendor/bundle/ruby/1.9.1/gems/json-1.7.3/lib/json/common.rb:155:inparse' /app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.1/lib/multi_json/engines/json_gem.rb:13:in decode' /app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.1/lib/multi_json.rb:65:indecode' /app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.7.0/lib/faraday/response/parse_json.rb:16:in parse' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:17:inon_complete' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:9:in block in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:62:inon_complete' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:8:in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:8:incall' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/request/url_encoded.rb:14:in call' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/connection.rb:203:inrun_request' /app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/connection.rb:90:in post' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:23:inblock in post' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:66:in call' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:66:inlog' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:22:in post' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/configuration.rb:16:instart_sphinx' /app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/tasks.rb:13:in block (2 levels) in <top (required)>' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:incall' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:in block in execute' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:ineach' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:in execute' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:158:inblock in invoke_with_call_chain' /usr/local/lib/ruby/1.9.1/monitor.rb:201:in mon_synchronize' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:151:ininvoke_with_call_chain' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:176:in block in invoke_prerequisites' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:174:ineach' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:174:in invoke_prerequisites' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:157:inblock in invoke_with_call_chain' /usr/local/lib/ruby/1.9.1/monitor.rb:201:in mon_synchronize' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:151:ininvoke_with_call_chain' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:144:in invoke' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:112:ininvoke_task' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in block (2 levels) in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:ineach' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in block in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:instandard_exception_handling' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:84:in top_level' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:62:inblock in run' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:in standard_exception_handling' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:59:inrun' /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/bin/rake:32:in <top (required)>' /app/vendor/bundle/ruby/1.9.1/bin/rake:19:inload' /app/vendor/bundle/ruby/1.9.1/bin/rake:19:in `'

Many Thanks.

  1. Support Staff 1 Posted by Pat Allan on 17 Jul, 2012 08:45 AM

    Pat Allan's Avatar

    Hi Alex

    Can you do the following please:

        heroku config:add VERBOSE_LOGGING=true
        heroku run rake fs:rebuild

    And let me know what the full output of that is?

    Also, try `heroku restart` to reboot your job workers to see if that helps.

    Cheers

    Pat

  2. 2 Posted by Alex Serdyuk on 17 Jul, 2012 11:26 PM

    Alex Serdyuk's Avatar

    Hi Pat,

    Attached is the verbose stack trace. The interesting bit is towards the very end, when Sphinx is trying to start it seemingly renders the 500 error page. Note that the strange behavior is started to occur when we've upgraded from wooden to ceramic. I've tried downgrade upgrade cycle again, but the problem persists.

    Thanks.

  3. Support Staff 3 Posted by Pat Allan on 18 Jul, 2012 10:00 AM

    Pat Allan's Avatar

    Hi Alex

    I think I've found the problem - a bug in flying-sphinx.com assigns already taken ports when switching plans. A fix is easy enough, but there seems to be an issue with Heroku at the moment that's getting in the way. I've contacted them and will hopefully have it resolved shortly.

    Cheers

    Pat

  4. Support Staff 4 Posted by Pat Allan on 18 Jul, 2012 11:24 AM

    Pat Allan's Avatar

    Okay, I've got the plan change fix in place, but Heroku's issue is stopping me from fixing your app directly. Can you downgrade/upgrade again, and then let me know how the fs:rebuild run goes?

  5. Support Staff 5 Posted by Pat Allan on 18 Jul, 2012 01:12 PM

    Pat Allan's Avatar

    Okay, got that issue sorted, fixed up your port numbers, started Sphinx - you should be fine now (no need to run fs:rebuild or change plans). Let me know if I'm wrong.

  6. 6 Posted by Alex Serdyuk on 18 Jul, 2012 11:44 PM

    Alex Serdyuk's Avatar

    Hi Pat,

    Many thanks for fixing the problem! I've run fs:rebuild and that was smooth after that! I actually have one more app that I've recently upgraded from wooden to ceramic, but I haven't deployed to/tested it yet. Is this fix you've done is on per-app basis? Does anything needs to be done about the second app?

    Kind Regards.

  7. Support Staff 7 Posted by Pat Allan on 19 Jul, 2012 09:01 AM

    Pat Allan's Avatar

    It's a per-app fix, but either let me know the app's name, or just downgrade/upgrade again, and it should be fine.

    Cheers

    Pat

  8. 8 Posted by Alex Serdyuk on 23 Jul, 2012 03:31 AM

    Alex Serdyuk's Avatar

    Hi Pat,

    I believe we also have what appears to be the same issue on another app:
    http://jedo-staging.herokuapp.com

    I've tried upgrade/downgrade cycle from wooden to ceramic but it hasn't worked. Could you please fix this app, as we really need it for our UAT.

    Many Thanks,
    -Alex

  9. Support Staff 9 Posted by Pat Allan on 23 Jul, 2012 07:51 AM

    Pat Allan's Avatar

    Hi Alex

    Just fixed this app up - let me know if you hit any further problems.

    Cheers

    Pat

  10. 10 Posted by Alex Serdyuk on 25 Jul, 2012 02:46 AM

    Alex Serdyuk's Avatar

    Hi Pat,

    I'm still having some issues with "jedo-staging" app, in particular starting Sphinx. Could you please have a look? I've attached the verbose log for you to have a look.

    Many thanks,
    -Alex

  11. Support Staff 11 Posted by Pat Allan on 25 Jul, 2012 11:33 AM

    Pat Allan's Avatar

    Hi Alex

    Sorry about that, turns out there was a bug on my server which was only cropping up with the 0.7 flying-sphinx release. I've just fixed it, so things should be smoother now for jedo-staging. If it's not, let me know.

    Cheers

    --
    Pat

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac