Hi,
I have the following error when trying to use searchd.
FlyingSphinx::FlagAsDeletedJob failed with
Riddle::ResponseError: searchd error (status: 1): invalid client
key '209a2e7a264af3393:9167e7e5fba407f71' - 2 failed
attempts
The error started happening when I swtiched from wooden to ceramic in order to use deltas. The version of flying-sphinx I'm using is the newest 0.7. Any help would be appreciated.
Also, It seems like it stops and indexes fine, but when I start up I get an error (stack trace with --trace):
/app/vendor/bundle/ruby/1.9.1/gems/json-1.7.3/lib/json/common.rb:155:in
parse'
/app/vendor/bundle/ruby/1.9.1/gems/json-1.7.3/lib/json/common.rb:155:inparse'
/app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.1/lib/multi_json/engines/json_gem.rb:13:in
decode'
/app/vendor/bundle/ruby/1.9.1/gems/multi_json-1.0.1/lib/multi_json.rb:65:indecode'
/app/vendor/bundle/ruby/1.9.1/gems/faraday_middleware-0.7.0/lib/faraday/response/parse_json.rb:16:in
parse'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:17:inon_complete'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:9:in
block in call'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:62:inon_complete'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:8:in
call'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/response.rb:8:incall'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/request/url_encoded.rb:14:in
call'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/connection.rb:203:inrun_request'
/app/vendor/bundle/ruby/1.9.1/gems/faraday-0.7.3/lib/faraday/connection.rb:90:in
post'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:23:inblock
in post'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:66:in
call'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:66:inlog'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/api.rb:22:in
post'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/configuration.rb:16:instart_sphinx'
/app/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-0.7.0/lib/flying_sphinx/tasks.rb:13:in
block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:incall'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:in
block in execute'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:ineach'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:200:in
execute'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:158:inblock
in invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/monitor.rb:201:in mon_synchronize'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:151:ininvoke_with_call_chain'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:176:in
block in invoke_prerequisites'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:174:ineach'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:174:in
invoke_prerequisites'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:157:inblock
in invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/monitor.rb:201:in mon_synchronize'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:151:ininvoke_with_call_chain'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:144:in
invoke'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:112:ininvoke_task'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in
block (2 levels) in top_level'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:ineach'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in
block in top_level'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:instandard_exception_handling'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:84:in
top_level'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:62:inblock
in run'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:129:in
standard_exception_handling'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:59:inrun'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/bin/rake:32:in
<top (required)>'
/app/vendor/bundle/ruby/1.9.1/bin/rake:19:inload'
/app/vendor/bundle/ruby/1.9.1/bin/rake:19:in `'
Many Thanks.
#1 Posted by Pat Allan Support staff on 17 Jul, 2012 08:45 AM
Hi Alex
Can you do the following please:
heroku config:add VERBOSE_LOGGING=true
heroku run rake fs:rebuild
And let me know what the full output of that is?
Also, try `heroku restart` to reboot your job workers to see if that helps.
Cheers
Pat
#2 Posted by Alex Serdyuk on 17 Jul, 2012 11:26 PM
Hi Pat,
Attached is the verbose stack trace. The interesting bit is towards the very end, when Sphinx is trying to start it seemingly renders the 500 error page. Note that the strange behavior is started to occur when we've upgraded from wooden to ceramic. I've tried downgrade upgrade cycle again, but the problem persists.
Thanks.
#3 Posted by Pat Allan Support staff on 18 Jul, 2012 10:00 AM
Hi Alex
I think I've found the problem - a bug in flying-sphinx.com assigns already taken ports when switching plans. A fix is easy enough, but there seems to be an issue with Heroku at the moment that's getting in the way. I've contacted them and will hopefully have it resolved shortly.
Cheers
Pat
#4 Posted by Pat Allan Support staff on 18 Jul, 2012 11:24 AM
Okay, I've got the plan change fix in place, but Heroku's issue is stopping me from fixing your app directly. Can you downgrade/upgrade again, and then let me know how the fs:rebuild run goes?
#5 Posted by Pat Allan Support staff on 18 Jul, 2012 01:12 PM
Okay, got that issue sorted, fixed up your port numbers, started Sphinx - you should be fine now (no need to run fs:rebuild or change plans). Let me know if I'm wrong.
#6 Posted by Alex Serdyuk on 18 Jul, 2012 11:44 PM
Hi Pat,
Many thanks for fixing the problem! I've run fs:rebuild and that was smooth after that! I actually have one more app that I've recently upgraded from wooden to ceramic, but I haven't deployed to/tested it yet. Is this fix you've done is on per-app basis? Does anything needs to be done about the second app?
Kind Regards.
#7 Posted by Pat Allan Support staff on 19 Jul, 2012 09:01 AM
It's a per-app fix, but either let me know the app's name, or just downgrade/upgrade again, and it should be fine.
Cheers
Pat
#8 Posted by Alex Serdyuk on 23 Jul, 2012 03:31 AM
Hi Pat,
I believe we also have what appears to be the same issue on another app:
http://jedo-staging.herokuapp.com
I've tried upgrade/downgrade cycle from wooden to ceramic but it hasn't worked. Could you please fix this app, as we really need it for our UAT.
Many Thanks,
-Alex
#9 Posted by Pat Allan Support staff on 23 Jul, 2012 07:51 AM
Hi Alex
Just fixed this app up - let me know if you hit any further problems.
Cheers
Pat
#10 Posted by Alex Serdyuk on 25 Jul, 2012 02:46 AM
Hi Pat,
I'm still having some issues with "jedo-staging" app, in particular starting Sphinx. Could you please have a look? I've attached the verbose log for you to have a look.
Many thanks,
-Alex
#11 Posted by Pat Allan Support staff on 25 Jul, 2012 11:33 AM
Hi Alex
Sorry about that, turns out there was a bug on my server which was only cropping up with the 0.7 flying-sphinx release. I've just fixed it, so things should be smoother now for jedo-staging. If it's not, let me know.
Cheers
--
Pat