Thinking of migrating to Sidekiq with TS3

lephyrius's Avatar


17 Jan, 2013 06:14 PM

I feel that Sidekiq is the way to go in the future.
I have recently migrated to TS3.
What branch should I use?

Is it even possible to run FlyingSphinx + TS3 with Sidekiq? Or is it too early?

  1. Support Staff 1 Posted by Pat Allan on 18 Jan, 2013 12:04 AM

    Pat Allan's Avatar

    I've been using it over a month now with TS v3 on a production site (66k delta requests), and it works fine. You'll need to use the sidekiq branch of my ts-resque-delta fork though (the original ts-sidekiq-delta was forked from ts-resque-delta, hence Github got a bit confused with the naming):

    gem 'ts-sidekiq-delta',
      git: 'git://',
      ref: 'db4f9d2a5c'
  2. 2 Posted by lephyrius on 16 Mar, 2013 08:14 AM

    lephyrius's Avatar

    I finally got time to transition to Sidekiq.
    One question is:
    So this is a branch of ResqueDelta.
    In my delta arguement what should I use:
    FlyingSphinx::ResqueDelta or is there a:
    FlyingSphinx::SidekiqDelta or should I use:

    Im moving from DelayedDelta. So this is a bit new to me.

  3. Support Staff 3 Posted by Pat Allan on 16 Mar, 2013 09:15 AM

    Pat Allan's Avatar

    It's ThinkingSphinx::Deltas::SidekiqDelta. This will work locally and on Flying Sphinx.

  4. 4 Posted by lephyrius on 23 Mar, 2013 04:17 PM

    lephyrius's Avatar

    I tried to to do it wtih this code:

    ThinkingSphinx::Index.define :game, with: :active_record, delta: true do
      #if Rails.env.production?
      #  set_property delta: FlyingSphinx::DelayedDelta
      #  set_property delta: :delayed
      set_property delta: ThinkingSphinx::Deltas::SidekiqDelta
    Using sidekq 2.8.0 it work perfectly in local development but not at all for flying sphinx. It just contacts the flying sphinx server from the web server thread not running the sidekiq worker at all. It's like iits not being enqueued at all.
    Using Heroku.
  5. 5 Posted by luki3k5 on 10 Apr, 2013 12:46 PM

    luki3k5's Avatar

    Hey Guys,

    I've just hit the same problem, is there any new developments?


  6. Support Staff 6 Posted by Pat Allan on 10 Apr, 2013 03:35 PM

    Pat Allan's Avatar

    Hi Lukasz

    Can you share your index definition here?



  7. 7 Posted by Lukasz on 10 Apr, 2013 05:37 PM

    Lukasz's Avatar

    Hi Pat,

    Perhaps it makes sense to tell you what my plan is - I wish we could move away from delayed_job completely, so for now I am using sidekiq for everything else like images processing and emails. I wish I could drop this last worker of delayed_job that we use
    for getting the delta indexes.

    So after this small introduction here is the code, gem file and issue that keeps poping up:

    define_index do
        indexes :store_id
        indexes :category_id
        indexes :brand_id
        indexes :user_id
        indexes :state
        indexes store(:name),    :as => :store_name,    :sortable => true
        indexes brand(:name),    :as => :brand_name,    :sortable => true
        indexes category(:name), :as => :category_name, :sortable => true
        indexes,       :as => :tags
        join store
        has "published_at"
        has "RADIANS(stores.latitude)",  :as => :latitude,  :type => :float    
        has "RADIANS(stores.longitude)", :as => :longitude, :type => :float
        group_by 'stores.latitude', 'stores.longitude'
        if Rails.env.production? || Rails.env.staging?
          set_property :delta => ThinkingSphinx::Deltas::SidekiqDelta

    Here is my gemfile:

    source ''
    ruby '1.9.3'
    gem 'foreman'
    gem 'rails', '3.2.12'
    gem "activeadmin"
    gem "airbrake", '3.1.6'
    gem 'aws-sdk', '~> 1.3.4'
    gem 'bcrypt-ruby', '~> 3.0.0'
    gem 'carrierwave'
    gem 'carrierwave_backgrounder'
    gem 'country-select'
    gem 'dalli'
    gem 'draper'
    gem "devise", "2.0.4"
    gem 'delayed_job_active_record'
    gem 'impressionist' 
    gem 'jbuilder', '1.0.2'#'0.8.0'
    gem 'jquery-rails'
    gem 'kaminari'
    gem 'koala'
    gem 'less'
    gem 'memcachier'
    gem "nokogiri"
    gem 'newrelic_rpm'
    gem 'pg'
    gem 'pusher'
    gem 'puma'
    gem 'riddle', :git => "git://", :ref => "5b4acc73aff52dd9ea919503a7ce33adcf6431c3"
    gem 'rmagick'
    gem 'foreigner'
    gem "fog", "~> 1.3.1"
    gem 'google-spreadsheet-ruby'
    gem 'geokit' # this should allow for easy finding of addresses for stores. 
    gem "haml"
    gem 'sidekiq'
    gem 'state_machine'
    gem 'sinatra', require: false
    gem 'slim'
    gem "typhoeus"
    gem 'twilio-ruby'
    gem "urbanairship"
    gem "yajl-ruby" # better jbuilder performance? 
    gem 'thinking-sphinx'
    gem 'flying-sphinx'
    gem 'mysql2'
    gem 'ts-sidekiq-delta',
      git: 'git://',
      ref: 'db4f9d2a5c'
    group :assets do
      gem 'angularjs-rails'
      gem 'sass-rails'
      gem 'compass-rails'
      gem 'coffee-rails', '~> 3.2.1'
      gem 'uglifier', '>= 1.0.3'
      gem "twitter-bootstrap-rails"
      gem "less-rails"
      gem 'turbo-sprockets-rails3'
    gem "therubyracer"
    group :test do
      gem 'webmock'
      gem 'rspec'#, '~> 2.10.0'
      gem 'rspec-rails'
      gem 'shoulda-matchers'
    group :test, :development do
      gem 'factory_girl_rails', require: false
      gem 'awesome_print'
      gem "capybara"
      gem 'guard-rspec'
      gem 'spork'
      gem 'rb-fsevent'
      gem 'guard-spork'
      gem 'growl'
      gem 'fivemat'
      gem 'timecop'
      gem "parallel_tests"
      gem 'jasminerice'
      gem 'guard-jasmine'

    I had some issues with running this locally first there was mysql2 gem missing (not sure if this will work on heroku) and now I have this one:

    /Users/luki3k5/.rvm/gems/ruby-1.9.3-p327/gems/flying-sphinx-0.8.5/lib/flying_sphinx/heroku_shared_adapter.rb:1:in `<top (required)>': uninitialized constant ThinkingSphinx::PostgreSQLAdapter (NameError)
        from /Users/luki3k5/.rvm/gems/ruby-1.9.3-p327/gems/flying-sphinx-0.8.5/lib/flying_sphinx.rb:21:in `<top (required)>'

    Thanks in advance!

  8. 8 Posted by Lukasz on 10 Apr, 2013 05:44 PM

    Lukasz's Avatar

    Ahh I've just found this thread:

    Okay then, I will continue with this, since obviously I have the old index files.

    Sorry for too hasty message.

  9. Support Staff 9 Posted by Pat Allan on 11 Apr, 2013 12:26 AM

    Pat Allan's Avatar

    Yeah, my fork of ts-sidekiq-delta is Thinking Sphinx v3 only.

    If you hit any other problems, do let me know.



  10. 10 Posted by Lukasz on 11 Apr, 2013 10:36 AM

    Lukasz's Avatar

    Hey Pat,

    I had some problems with installation and then indexing but following all the threads in support I managed to get over them, however now I hit the wall again and would appriciate any hint.

    So here is my Issue:

    (sphinxql: syntax error, unexpected USERVAR, expecting IDENT (or 5 other tokens) near '@geodist BETWEEN 0.0 AND 10000000.0 AND sphinx_deleted = 0 ORDER BY published_at DESC LIMIT 0, 10; SHOW META'):

    and here is an index for Item class:

    ThinkingSphinx::Index.define :item, with: :active_record, delta: true do
      indexes :store_id
      indexes :category_id
      indexes :brand_id
      indexes :user_id
      indexes :state
      indexes store(:name),    :as => :store_name,    :sortable => true
      indexes brand(:name),    :as => :brand_name,    :sortable => true
      indexes category(:name), :as => :category_name, :sortable => true
      indexes,       :as => :tags
      join store
      has "published_at"
      has "RADIANS(stores.latitude)",  :as => :latitude,  :type => :float    
      has "RADIANS(stores.longitude)", :as => :longitude, :type => :float
      group_by 'stores.latitude', 'stores.longitude'

    and here is how I used to use sphinx (prior update) and what I believe is now in a wrong way:

         items = [d2r(latitude), d2r(longitude)], 
                                             per_page: VIEWS_FEED_ITEMS_PER_PAGE,
                                             page: page,
                                             conditions: { :state => "published" },
                                             with: {"@geodist" => 0.0..DEFAULT_RADIUS },
                                             order: "published_at DESC")

    I have 'ThinkingSphinx::Configuration::Defaults::PANES << ThinkingSphinx::Panes::DistancePane
    ' setup inside the initializer but I don't think this has anything to do with it as if I understood documentation this pane gets auto-injected when 'geo' attribute is passed in.

    Please let me know what am I missing, all hints are always greatly appreciated.


  11. Support Staff 11 Posted by Pat Allan on 11 Apr, 2013 02:49 PM

    Pat Allan's Avatar

    Hi Lukasz

    The issue here is that SphinxQL just requires geodist, not @geodist - it's no longer a special attribute, but a generated one. If you remove the @ (and if you like, change it to the symbol (e.g.: :geodist) when using it in a filter), it should work fine.



  12. 12 Posted by mauro on 11 Apr, 2013 07:11 PM

    mauro's Avatar

    hi pat I just migrate TS 2 to TS3 as said in the readme but when I tried to run the rake for make config file i get his

    bundle exec rake ts:configure --trace
    ** Invoke ts:configure (first_time)
    ** Invoke environment (first_time)
    ** Execute environment
    ** Execute ts:configure
    Generating configuration to /Users/chebyte/Dropbox/Rails/agencieshq/config/development.sphinx.conf
    rake aborted!
    undefined method `type' for nil:NilClass
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/attribute/type.rb:64:in `type_from_database'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/attribute/type.rb:17:in `type'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/attribute.rb:4:in `type'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/attribute/sphinx_presenter.rb:30:in `sphinx_type'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/attribute/sphinx_presenter.rb:18:in `collection_type'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/sql_source.rb:113:in `block in prepare_for_render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/sql_source.rb:110:in `each'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/sql_source.rb:110:in `prepare_for_render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/active_record/sql_source.rb:65:in `render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration/index.rb:29:in `block in render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration/index.rb:29:in `collect'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration/index.rb:29:in `render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/core/index.rb:48:in `render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration.rb:39:in `block in render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration.rb:39:in `collect'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/riddle-1.5.5/lib/riddle/configuration.rb:39:in `render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/configuration.rb:78:in `render'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/configuration.rb:84:in `block in render_to_file'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:35:in `open'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/open-uri.rb:35:in `open'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/configuration.rb:84:in `render_to_file'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/rake_interface.rb:4:in `configure'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/thinking-sphinx-3.0.2/lib/thinking_sphinx/tasks.rb:4:in `block (2 levels) in <top (required)>'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:228:in `call'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:228:in `block in execute'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:223:in `each'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:223:in `execute'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:166:in `block in invoke_with_call_chain'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:159:in `invoke_with_call_chain'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/task.rb:152:in `invoke'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:143:in `invoke_task'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:in `block (2 levels) in top_level'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:in `each'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:101:in `block in top_level'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:110:in `run_with_threads'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:95:in `top_level'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:73:in `block in run'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:160:in `standard_exception_handling'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/lib/rake/application.rb:70:in `run'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.0.3/bin/rake:33:in `<top (required)>'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/bin/rake:23:in `load'
    /Users/chebyte/Developer/.rbenv/versions/1.9.3-p194/bin/rake:23:in `<main>'
    Tasks: TOP => ts:configure
  13. 13 Posted by Laurent Gorse on 19 Apr, 2013 02:18 AM

    Laurent Gorse's Avatar

    Hey everyone,

    Joining in as this is the only conversation out there about TS3 + Sidekiq

    I'm having an issue where when I create a model instance, the db rolls back right away with the error: 'undefined method 'toggle' for nil:NilClass'

    I think this has to do with how the delta is declared when the model instance is created.

    Here is the line that creates the instance:

    @restaurant = Restaurant.find_or_create_by_google_id(venue[:google_id], final_restaurant_attributes(venue))

  14. 14 Posted by lephyrius on 25 Apr, 2013 04:43 PM

    lephyrius's Avatar

    I have begun to get these warnings:
    2013-04-25T16:36:34Z 5300 TID-ovs5uycmo WARN: ThinkingSphinx::Deltas::SidekiqDelta::DeltaJob - :timeout is unsafe and support has been removed from Sidekiq, see for details 2013-04-25T16:36:34Z 5300 TID-ovs5uycmo WARN: ThinkingSphinx::Deltas::SidekiqDelta::FlagAsDeletedJob - :timeout is unsafe and support has been removed from Sidekiq, see for details Hmmm...
    This is probably just a problem with sidekiq deltas but I cannot report an issue on a branch.

  15. Support Staff 15 Posted by Pat Allan on 28 Apr, 2013 05:29 PM

    Pat Allan's Avatar

    This is odd, as there's nothing in my fork of ts-sidekiq-delta that mentions timeouts. Do you have any other Sidekiq code in place (default settings of some sort) that sets timeouts?

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac