Connection to 127.0.0.1 on 9312 failed.

Jay Liu

05 Sep, 2015 12:34 AM

When attempting to run a search using ThinkingSphinx#search, the following error pops up when the template tries to render:

Connection to 127.0.0.1 on 9312 failed. Connection refused - connect(2)) on line #21 of app/views/searches/show.html.haml

Why is there a connection attempt on 127.0.0.1, is something causing the environment variables that are set up for FlyingSphinx to not be read?

This application is hosted on Heroku, with the following:
Rails 2.3.15
thinking-sphinx 1.4.10
flying-sphinx 0.6.1

Admittedly the gems are probably outdated - but I'm looking to put out this fire until the application as a whole can be updated.

  1. #1 Posted by Pat Allan Support staff on 05 Sep, 2015 12:39 AM

    Hi Jay

    Did this work previously? Can you let me know what the name of the app is? Also, I'd highly recommend at least updating to thinking-sphinx 1.5.0 and flying-sphinx 1.2.0 (they should work together).

    Kind regards,

    Pat

  2. #2 Posted by Jay Liu on 05 Sep, 2015 03:40 AM

    Hi Pat,

    I'm just being brought on to the project to replace the previous developer. The name of the app is projectalert - accessible at http://projectalert.com. As far as I know it was operational previously - at the very least prior to last evening when a small update was pushed up (just replacing some image files).

    I'll go ahead and update those gems - which should also necessitate a Ruby version upgrade since it is currently running on 1.8.7.

    I'll let you know once I've been able to do this, thanks!

  3. #3 Posted by Pat Allan Support staff on 05 Sep, 2015 04:44 AM

    Both of those gem versions should work on 1.8.7, if that makes things easier (well, until Heroku forces you onto Cedar-14, where 1.9.3 is the minimum version of Ruby supported).

  4. #4 Posted by Jay Liu on 08 Sep, 2015 07:29 AM

    Hi Pat,

    I was able to get the gems for thinking-sphinx and flying-sphinx updated to 1.5 and 1.2, respectively. Unfortunately I'm running into the same issue and in addition, when I try to run `heroku run rake fs:rebuild` I'm getting the following error:

    Pusher : error : #<NoMethodError: undefined method `read_nonblock' for #<OpenSSL::SSL::SSLSocket:0x7f9d25f5bcd8>>

    I've included openssl-nonblock in my Gemfile as well, is there any guidance you can offer?

    Thanks!

  5. #5 Posted by Jay Liu on 08 Sep, 2015 07:35 AM

    In addition, I'm getting the same error with all of the legacy rake fs:* commands.

    Thanks for your help.

  6. #6 Posted by Pat Allan Support staff on 08 Sep, 2015 06:18 PM

    Just to confirm: you're still seeing the read_nonblock error with the openssl-nonblock in place? This is certainly unexpected. Is adding me to the project to debug directly an option? (My Heroku account is pat at freelancing-gods dot com). If it's not, that's completely understandable… I'll ponder further about what could be the cause…

  7. #7 Posted by Pat Allan Support staff on 08 Sep, 2015 06:23 PM

    Also, it's worth noting that the daemon is running on Flying Sphinx, and indexing calls have been coming through daily, so it feels like this is very much an issue with the app and its interaction with the Thinking Sphinx and Flying Sphinx gems. What's in your config/sphinx.yml file for the relevant environment (if the file doesn't exist, that's fine)? And are there any initialisers mucking about with Thinking Sphinx settings?

  8. #8 Posted by Jay Liu on 08 Sep, 2015 06:30 PM

    Yes, I'm still getting the read_nonblock error with the openssl-nonblock in place. I just tried to run rake fs:index and I got the following output:

    Executing Action: configure
  SQL (0.7ms)   SET client_min_messages TO 'panic'
  SQL (0.7ms)   SET standard_conforming_strings = on
  SQL (0.6ms)   SET client_min_messages TO 'notice'
rake aborted!
can't convert nil into String

    But the other actions, like :rebuild and :start both result in the non_block error.

    I totally agree with you about the app and its interaction, I'm just not sure what to change to remedy this situation. The config/sphinx.yml file is actually in .gitignore, so I'm not 100% sure what's going on with it in the production environment. From what I got from the previous developer it has the following:

    production:
  <<: *default
  config_file: config/production.sphinx.conf
  pid_file: log/searchd.production.pid
  searchd_file_path: db/sphinx/production/
  version: 1.10-beta

    As far as adding you to debug, I may have to ask the other project staff since it is still in production.

  9. #9 Posted by Jay Liu on 08 Sep, 2015 06:34 PM

    Also, I don't think that there's any initializers messing with Thinking Sphinx - at least I couldn't find any references to it in the project aside from the #search method calls and the indices.

  10. #10 Posted by Pat Allan Support staff on 08 Sep, 2015 07:06 PM

    Appreciate those details - nothing's jumping out as being obviously wrong. Those settings in config/sphinx.yml shouldn't have an impact (beyond the version, but that's fine, that one is useful).

    With the fs:index call failing - can you run it with --trace and let me know what the full output is?

  11. #11 Posted by Jay Liu on 08 Sep, 2015 07:09 PM

    Here's the full output:

    Running `rake fs:index --trace` attached to terminal... up, run.1394
(in /app)
** Invoke fs:index (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute fs:index
Executing Action: configure
  SQL (0.8ms)   SET client_min_messages TO 'panic'
  SQL (0.7ms)   SET standard_conforming_strings = on
  SQL (0.6ms)   SET client_min_messages TO 'notice'
rake aborted!
can't convert nil into String
/app/vendor/bundle/ruby/1.8/gems/multipart-post-2.0.0/lib/composite_io.rb:91:in `basename'
/app/vendor/bundle/ruby/1.8/gems/multipart-post-2.0.0/lib/composite_io.rb:91:in `initialize'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/gzipped_hash.rb:31:in `new'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/gzipped_hash.rb:31:in `gzip'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/gzipped_hash.rb:11:in `to_gzipped_hash'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/gzipped_hash.rb:11:in `each'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/gzipped_hash.rb:11:in `to_gzipped_hash'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/configuration_options.rb:19:in `gzipped_files_hash'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/configuration_options.rb:3:in `to_hash'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/controller.rb:20:in `configure'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:46:in `configure'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:29:in `send'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:29:in `run'
/app/vendor/bundle/ruby/1.8/gems/activesupport-2.3.15/lib/active_support/dependencies.rb:184:in `all?'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:27:in `each'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:27:in `all?'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/cli.rb:27:in `run'
/app/vendor/bundle/ruby/1.8/gems/flying-sphinx-1.2.0/lib/flying_sphinx/tasks.rb:4
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
/app/vendor/ruby-1.8.7/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
/app/vendor/bundle/ruby/1.8/gems/rake-0.8.7/bin/rake:31
/app/vendor/bundle/ruby/1.8/bin/rake:23:in `load'
/app/vendor/bundle/ruby/1.8/bin/rake:23

  12. #12 Posted by Pat Allan Support staff on 08 Sep, 2015 07:13 PM

    Can you see if switching to flying-sphinx 1.0.0 helps at all?

  13. #13 Posted by Jay Liu on 08 Sep, 2015 08:39 PM

    Hey Pat,

    I'm running into the same error with flying-sphinx 1.0.0 - both the being unable to search (the original error) and getting the NoMethodError on read_nonblock.

    I've added you as a collaborator on the project, hopefully that will help in diagnosing the issue, let me know if there's anything I can help with / if you need more information.

  14. #14 Posted by Pat Allan Support staff on 08 Sep, 2015 10:38 PM

    I've no idea why this is the case, but it seems as if Flying Sphinx is not being required, even though it's in the Gemfile and config/environment.rb.

    One way to work around this is to add an initializer that just has require 'flying_sphinx' and see if that helps? I'm hoping that resolves the wrong Sphinx address/port issue.

    As for the openssl_nonblock - can you try putting that gem above both of the Sphinx gems in both the Gemfile and config/environment.rb? Of course, perhaps its not being required either - so it wouldn't hurt adding a require statement for it to the same initializer, before the one for flying_sphinx.

    I've not made any commits to test these changes on the app itself, but when I run heroku run bundle exec ./script/console, both require 'flying_sphinx' and require 'openssl_nonblock' return true, indicating they're not already loaded.

  15. #15 Posted by Jay Liu on 08 Sep, 2015 11:32 PM

    I just put in the changes you recommended and it looks like there's some progress. I'm no longer seeing the connection error, but when the search is being executed I'm seeing:

    Sphinx Caught Sphinx exception: undefined methodcollect' for nil:NilClass (0 tries left)`

    I'm kind of assuming this is because I still need to run rake fs:rebuild on the server, which I currently have going now. If that's not the case let me know, either way I'll let you know what I get after the rebuild finishes.

  16. #16 Posted by Pat Allan Support staff on 08 Sep, 2015 11:33 PM

    I would presume it's related to the rebuild that you're currently running. Let me know how it's behaving once the rebuild is finished :)

  17. #17 Posted by Jay Liu on 09 Sep, 2015 01:14 AM

    It looks like some of the tasks I'm trying to run are timing out. I started running rake fs:rebuild and it seemed like it wasn't moving or doing anything (about an hour). I aborted the task and tried both rake fs:index, rake fs:start and even rake fs:stop and both returned this output:

    Action timed out. If this is happening regularly, please contact Flying Sphinx support: http://support.flying-sphinx.com

    Both had the 'Executing Action: *' and 'Action Finished: *` preceding the time out message.

    Thanks again for all of your help, Pat!

  18. #18 Posted by Pat Allan Support staff on 09 Sep, 2015 01:31 AM

    So, since flying-sphinx 1.0, websockets are used instead of polling to track when a task has finished. Annoyingly, it looks like it’s flaky with 1.8.7, even with openssl_nonblock (it would generally work for me in the past, but now it seems almost never. *sigh*)

    So, can you try downgrading to flying-sphinx 0.7.1 (from memory, the 0.8.x releases weren’t 1.8.7-friendly) and see how things go from there?

  19. #19 Posted by Jay Liu on 09 Sep, 2015 05:29 AM

    Pat, you are amazing. It's back into working order. I'm working on an upgrade of this application at this point anyway - but the project wanted this bit operational for the time being.

    Thanks for all of your help.

  20. #20 Posted by Pat Allan Support staff on 09 Sep, 2015 05:31 AM

    I’m very glad we got there - shame about the frustrating bugs along the way (especially the gems not being required - that is super odd!)

    Good luck with the upgrades :)

Reply to this discussion Internal reply only staff will see it

Markdown

Attaching KB article:

» »

Read more…

Attach files

Up to 10MB each.

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles