Error on Heroku for Delta Indexing when trying to rebuild index

alex

17 Dec, 2012 10:08 AM

Please note: details have been edited with the correct error. Previous error was a mistake in the model file.

Hello, this error is related to the one we were having trying to run FS on Heroku using Padrino. Following the instructions for delta indexing, when I try to rebuild the Sphinx config via this command:

heroku run padrino rake ts:rebuild (or fs:rebuild)

I get the following error:

Generating Configuration to /app/config/production.sphinx.conf
rake aborted!
uninitialized constant FlyingSphinx::ResqueDelta

Tasks: TOP => thinking_sphinx:index

We are not using Rails at all, just Padrino.

For Delta Indexing I did the following:

  1. added a a delta column to the model and set it to true by default
  2. added this line to our define_index:
    if Rails.env.production? set_property :delta => FlyingSphinx::ResqueDelta end
  3. added the gem file: gem 'ts-resque-delta'
  4. added the relations code to trigger delta index when they change as per this page: http://pat.github.com/ts/en/deltas.html

Thanks

  1. #1 Posted by alex on 17 Dec, 2012 10:59 AM

    Looking at the ts-resque-delta/resque_delta.rb file, I see these 2 lines:

    require 'thinking_sphinx/deltas/resque_delta/flag_as_deleted_set'
require 'thinking_sphinx/deltas/resque_delta/index_utils'

    But in my thinking-sphinx gem, the delta folder (.../vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/deltas/) there is no such file as resque_delta. Only default_delta.

    Maybe this is part of the problem?

  2. #2 Posted by Pat Allan Support staff on 17 Dec, 2012 12:59 PM

    Hi Alex

    Sounds like you're doing everything right, but I guess that gem hasn't been built in a way that it'll work without Rails. Can you run the rake task with the --trace argument and let me know what the stacktrace is?

    Cheers

    Pat

  3. #3 Posted by Pat Allan Support staff on 17 Dec, 2012 01:00 PM

    Also - those require statements are for files within the ts-resque-delta gem, not within thinking-sphinx.

  4. #4 Posted by alex on 17 Dec, 2012 01:21 PM

    Sure thing Pat, following is the output.

    Also, I've kept your access to the app in Heroku if you want to try anything yourself.

    --------------

    MacBookPro:doghouseapi alexd$ heroku run padrino rake ts:rebuild --trace
    Running `padrino rake ts:rebuild --trace` attached to terminal... up, run.2463
    => Executing Rake ts:rebuild --trace ...
    ** [NewRelic][12/17/12 13:20:09 +0000 3b1b976b-df0a-424d-a820-2b03f442883d (2)] INFO : Dispatcher: sinatra
    ** [NewRelic][12/17/12 13:20:09 +0000 3b1b976b-df0a-424d-a820-2b03f442883d (2)] INFO : Application: doghouseapi
    ** [NewRelic][12/17/12 13:20:09 +0000 3b1b976b-df0a-424d-a820-2b03f442883d (2)] INFO : New Relic Ruby Agent 3.5.3 Initialized: pid = 2
    ** Invoke ts:rebuild (first_time)
    ** Invoke thinking_sphinx:rebuild (first_time)
    ** Invoke thinking_sphinx:app_env (first_time)
    ** Execute thinking_sphinx:app_env
    ** Execute thinking_sphinx:rebuild
    ** Invoke thinking_sphinx:index (first_time)
    ** Invoke thinking_sphinx:app_env
    ** Execute thinking_sphinx:index
    Generating Configuration to /app/config/production.sphinx.conf
    rake aborted!
    uninitialized constant FlyingSphinx::ResqueDelta
    /app/app/models/dvd.rb:92:in `block in <class:Dvd>'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/index/builder.rb:32:in `instance_eval'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/index/builder.rb:32:in `initialize'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/index/builder.rb:22:in `new'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/index/builder.rb:22:in `generate'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:207:in `add_sphinx_index'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:174:in `block in define_index'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:189:in `call'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:189:in `block in define_indexes'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:188:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/active_record.rb:188:in `define_indexes'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/configuration.rb:161:in `block in generate'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/configuration.rb:159:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/configuration.rb:159:in `generate'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/configuration.rb:175:in `build'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/tasks.rb:88:in `block (2 levels) in <top (required)>'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
    /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
    /app/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-2.0.13/lib/thinking_sphinx/tasks.rb:107:in `block (2 levels) in <top (required)>'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
    /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:176:in `block in invoke_prerequisites'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:174:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:174:in `invoke_prerequisites'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:157:in `block in invoke_with_call_chain'
    /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
    /app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
    /app/vendor/bundle/ruby/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/cli/rake.rb:15:in `init'
    /app/vendor/bundle/ruby/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/cli/base.rb:55:in `rake'
    /app/vendor/bundle/ruby/1.9.1/gems/thor-0.15.4/lib/thor/task.rb:27:in `run'
    /app/vendor/bundle/ruby/1.9.1/gems/thor-0.15.4/lib/thor/invocation.rb:120:in `invoke_task'
    /app/vendor/bundle/ruby/1.9.1/gems/thor-0.15.4/lib/thor.rb:275:in `dispatch'
    /app/vendor/bundle/ruby/1.9.1/gems/thor-0.15.4/lib/thor/base.rb:425:in `start'
    /app/vendor/bundle/ruby/1.9.1/gems/padrino-core-0.10.7/bin/padrino:9:in `<top (required)>'
    /app/bin/padrino:16:in `load'
    /app/bin/padrino:16:in `<main>'
    Tasks: TOP => thinking_sphinx:index

  5. #5 Posted by Pat Allan Support staff on 18 Dec, 2012 01:59 AM

    Hi Alex

    I just pushed a single commit to the app on Heroku - a manual require of 'flying_sphinx/resque_delta'. That seems to have done the trick, a full index works now (though it's worth noting that a full index takes 4 or 5 minutes now. May want to put a database index on the delta column to help keep the indexer queries speedy.

    If there's any further challenges, let me know.

    Cheers

    Pat

  6. #6 Posted by alex on 18 Dec, 2012 08:56 AM

    Thanks Pat I shall look at it,

    Couple of questions:

    1. You say a full index works now. Full index was always working, how about a delta index? Maybe I'm not understanding you (or Delta indexing) properly.

    2. the delta column does already have a database index as suggested by one of the readme's, it's been there since the beginning:

    KEY `delta` (`delta`),

    3. Is there a need to put a delta column into any of the related data models? For example, we have an actors junction table table that has more 2 million rows and gets updated when the delta index runs since I copied these lines from one of the readmes:

    # Sphinx delta indexing for the associations to work
      after_save :set_dvd_delta_flag
      after_destroy :set_dvd_delta_flag

      private

      def set_dvd_delta_flag
        dvd.delta = true
        dvd.save
      end

    Perhaps this is causing the delay? I actually put this into the actors model, not the junction table model because I don't have a file for it. I'm assuming ActiveRecord is smart enough to update the appropriate table...

    4 minutes seems excessive, I don't remember it taking this long before to do a full index.

  7. #7 Posted by Pat Allan Support staff on 22 Dec, 2012 12:22 PM

    Hi Alex

    Sorry, this slipped by. I should have responded sooner. To answer your questions:

    1. Delta indexing should be processed when you have a worker running. Have you trialled this at all yet? From what I can see, it looks like some jobs have been added to Resque/Redis.
    2. Deltas in associated models are not needed - indeed, unless Thinking Sphinx is indexing that model, it won't be used at all. Your callbacks take the right approach with updating the indexed model record. That won't be causing the long Sphinx indexing times though - if five minutes is too long, then it may be worth analysing the query generated by Thinking Sphinx (it'll be the same locally as it is on Heroku) and figuring out whether there's any slow joins and/or missing database indexes.

    Kind regards,

    Pat

  8. #8 Posted by alex on 28 May, 2013 03:59 PM

    Hi Pat, I see you have updated TS 3.x to 3.0.3. We had to put this into our Gem file because we are using Sinatra:

    gem 'thinking-sphinx', '3.0.1',
      :require => 'thinking_sphinx/sinatra',
      :git     => 'git://github.com/pat/thinking-sphinx.git',
      :ref     => '0d28f30634'
    group :production, :worker do
      gem 'flying-sphinx',
        :git     => 'git://github.com/flying-sphinx/flying-sphinx.git',
        :ref     => '0fec6ca612',
        :require => 'flying_sphinx/sinatra'
        gem 'ts-sidekiq-delta', git: 'git://github.com/pat/ts-resque-delta.git', branch: 'sidekiq'
    end

    As suggested by you. Can I now get rid of all the extra attributes and just use:

    gem 'thinking-sphinx'

    ?

    Thanks.

  9. #9 Posted by Pat Allan Support staff on 28 May, 2013 11:39 PM

    Hi Alex

    You can drop the git/ref settings for Thinking Sphinx and Flying Sphinx, but you need to keep them for ts-sidekiq-delta. So here's the end result:

    gem 'thinking-sphinx',  '~> 3.0.3',
  :require => 'thinking_sphinx/sinatra'
gem 'flying-sphinx',    '~> 1.0.0',
  :require => 'flying_sphinx/sinatra'
gem 'ts-sidekiq-delta',
  :git    => 'git://github.com/pat/ts-resque-delta.git',
  :branch => 'sidekiq',
  :ref    => '5bae80758c'

    Cheers

    Pat

  10. #10 Posted by alex on 29 May, 2013 08:50 AM

    Thanks.

  11. Pat Allan closed this discussion on 13 Jan, 2014 11:23 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

Recent Articles