heroku run fs:index taking not working

Ashwin

18 Jun, 2012 06:07 PM

Hi,

I deployed my new rails application recently on Heroku and I'm using flying sphinx addon for implementing search feature, I have added wooden version of flying-sphinx and when I try to index the index the command is not responding with anything and not completing.

The command I'm trying to run is

heroku run rake fs:index

and the application is deployed on cedar.

This is my settings:

Gemfile:

gem 'thinking-sphinx', '2.0.10', :require => 'thinking_sphinx'
gem 'flying-sphinx', '0.6.5'

config/sphinx.yml

development:
version: '1.10-beta'

production:
version: '1.10-beta'

When I run heroku run fs:index , it never indexes and hence the sphinx search is application is not working, Any help will be appreciated.

Thanks in advance.

  1. #1 Posted by Pat Allan Support staff on 18 Jun, 2012 06:16 PM

    Hi Ashwin

    What's your app's name? Knowing that will make it much easier for me to investigate on the server.

    Cheers

    Pat

  2. #2 Posted by Ashwin on 18 Jun, 2012 06:19 PM

    Hi Pat,

    The application name is ftcomp (ftcomp.herokuapp.com)

    Thanks,
    Ashwin

  3. #3 Posted by Ashwin on 18 Jun, 2012 06:24 PM

    Hi Pat,

    I found this new response which says the error below.

    heroku run rake fs:index

    Running rake fs:index attached to terminal... up, run.1

      Starting Index Request
  rake aborted!
  end of file reached

     Tasks: TOP => fs:index

    Also when I check the heroku logs from my application it says like below.

    2012-06-18T18:09:32+00:00 heroku[run.1]: Awaiting client
    2012-06-18T18:09:32+00:00 heroku[run.1]: Starting process with command bundle exec rake fs:index
    2012-06-18T18:09:32+00:00 heroku[run.1]: State changed from starting to up
    2012-06-18T18:10:08+00:00 heroku[run.1]: Process exited with status 1
    2012-06-18T18:10:08+00:00 heroku[run.1]: State changed from up to complete

    Please check and let me know if I have to change any configuration in my application.

    Thanks,
    Ashwin

  4. #4 Posted by Pat Allan Support staff on 18 Jun, 2012 06:27 PM

    Hi Ashwin

    It looks like you've added the FLYING_SPHINX_INGRESS variable, but I'm guessing you're using the standard Heroku shared database? If so, then you'll need to remove it:

    heroku config:remove FLYING_SPHINX_INGRESS

    If either of those guesses are wrong, can you let me know what database plan/add-on you're using, and the output from heroku config (well, the FLYING_SPHINX variables).

    Cheers

    Pat

  5. #5 Posted by Ashwin on 18 Jun, 2012 06:36 PM

    Hi Pat,

    Thanks for the quick response, I removed FLYING_SPHINX_INGRESS as you said, and tried to index again which still is not executing.

    This is the heroku config I have for flying sphinx,

    => 825ff0fea95686872
   => ec2-23-20-127-82.compute-1.amazonaws.com

    => 238f272cbe45d88a1

       => 9317

    I'm using shared database and I'm not using any database plan/add-on. Please let me know if you need complete config information of the application.

    Thanks,
    Ashwin

  6. #6 Posted by Ashwin on 18 Jun, 2012 06:39 PM

    Hi Pat,

    Were you able to see the trace of heroku config which I posted ? It doesn't look formatted in the comment I posted before.

    Thanks,
    Ashwin

  7. #7 Posted by Pat Allan Support staff on 18 Jun, 2012 06:41 PM

    Hi Ashwin

    Now it looks like there's something not quite right with your index definitions… can you provide your define_index blocks?

    Also, you can view the indexing output yourself via the dashboard:

        heroku addons:open flying_sphinx

    Cheers

    --
    Pat

  8. #8 Posted by Ashwin on 18 Jun, 2012 06:54 PM

    Hi Pat,

    These are my define_index blocks,

    define_index do

    indexes match1, :sortable => true
indexes permission, :sortable => true
indexes user_id, :sortable => true

    end

    define_index do

    indexes question, :sortable => true
indexes permission, :sortable => true
indexes user_id, :sortable => true

    end

    Here the permission is datatype string and user_id datatype is integer, I checked the logs of indexing in the addons log

    I find this error in the log, and nothing has been indexed.

    sql_range_query: ERROR: invalid input syntax for integer:

    Please let me know if this is anything to do with the application, when I searched this error I found that it has something to do with the postgres database.

    Thanks,

    Ashwin

  9. #9 Posted by Pat Allan Support staff on 18 Jun, 2012 07:26 PM

    How are you using the user_id field in each index? I'd recommend switching that to an attribute instead (which are always sortable), and can be integers natively in Sphinx - this should avoid the type issues.

    Pat

  10. #10 Posted by Ashwin on 19 Jun, 2012 09:26 AM

    Hi Pat,

    user_id is an attribute of the object wherever I have defined index and I'm using that attribute for extended match mode in my search methods, and I read in the documentation that integer datatypes are supported by flying sphinx and sortable too just like thinking sphinx. Can I know why is it not getting indexed and throwing the exception below ?

    indexing index 'matching_core'...
    ERROR: index 'matching_core': sql_range_query: ERROR: invalid input syntax for integer: ""

    If I cannot index user_id which is a attribute of the class, is there any other recommendations you suggest ? I need to use that user_id attribute for my search extended match modes.

    Thanks,
    Ashwin

  11. #11 Posted by Pat Allan Support staff on 19 Jun, 2012 09:44 AM

    Hi Ashwin

    Sorry if you already understand this, but just to be clear: Sphinx has fields and attributes, but they are not the same thing:
    http://freelancing-god.github.com/ts/en/sphinx_basics.html

    In Thinking Sphinx's index definitions, the 'indexes' method is for fields, the 'has' method is for attributes. If you're sorting by user_id, then it may be best to have it as an attribute, but that really depends on how else you're using it (can you show me the search calls?)

    That said, if you need user_id to be treated as a string, then your best option is to cast it as one:

        indexes "user_id::varchar", :as => :user_id, :sortable => true

    Cheers

    Pat

  12. #12 Posted by Ashwin on 19 Jun, 2012 10:07 AM

    Hi Pat,

    I tried defining the index as you said,

    indexes "user_id::varchar", :as => :user_id, :sortable => true

    But got this error below while indexing,

    column "matchings.user_id" must appear in the GROUP BY clause or be used in an aggregate function

    The search call in my application is like this

    @matchings = Matching.search "*#{params[:input_value]}* (@user_id #{current_user.id} | @permission #{'Public'})", :match_mode => :extended

    What I'm trying to do with is I need to fetch all the rows which has the input value in fields and either the user_id must be the current user or the permission should be public. Extended match mode is one way I found for doing this.

    Since I'm using user_id for checking in the extended match mode, I had to index that for searching.

    Thanks,
    Ashwin

  13. #13 Posted by Pat Allan Support staff on 19 Jun, 2012 10:14 AM

    Hi Ashwin

    Right. I guess that makes sense. So, you will need to add user_id to the GROUP BY clause manually - just add this to your index definition:

        group_by 'user_id'

    But also, given you're filtering internally, not by user input, I would personally change things so it's attribute-driven for the filter:

        has "permission = 'Public'", :as => :is_public, :type => :boolean
        has user_id

        group_by 'permission'

    And then searching:

        Matching.search params[:input_value],
          :star => true,
          :with => {:user_or_public => true},
          :sphinx_select => "*, IF(is_public OR user_id = #{current_user.id}, 1, 0) AS user_or_public"

    Granted, it is more detailed, but does provide that clearer separation. Let me know how whichever approach you choose goes.

    Pat

  14. #14 Posted by Ashwin on 19 Jun, 2012 12:22 PM

    Hi Pat,

    The latter approach worked for me and the search is up, thank you very much for your support.

    Currently I have wooden version of flying sphinx, will it be possible to use delta indexing with this version of flying sphinx ?

    Thanks,
    Ashwin

  15. #15 Posted by Pat Allan Support staff on 19 Jun, 2012 12:25 PM

    Great to hear it's working.

    As for deltas, I'm afraid it's not something I'll be adding to the wooden plan - from a resource perspective, it's just not feasible to have the increase in demand on those servers and keep everything reliable.

    Pat

  16. #16 Posted by nitin on 21 Aug, 2014 11:35 AM

    Hi Patt,

    I have an application(gamezone-next-dev.herokuapp.com) running on heroku. I ahve added flyings-phinx for searching. There is an issue i am facing with it. When i run rake fs:index, it just works fine and all of articles gets indexed.

    But whenever an article is created, it doesn't get indexed and says: 'sh: indexer not found'
    Can you please find what is the isuue

Reply to this discussion Internal reply only staff will see it

Markdown

Attaching KB article:

» »

Read more…

Attach files

Up to 10MB each.

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles