Hi Pat,
I am upgrading a project to the latest TS3 version everything is
working great, except the delayed deltas via Resque. I can trigger
the inline delta by setting :delta => true, but as
soon as I try the following:
ThinkingSphinx::Index.define :article, :with => :active_record, :delta => ::FlyingSphinx::ResqueDelta do
I don't get the delta jobs queued into the Redis queue. I have a feeling the ts-resque-delta gem is not updated to catch the new calls and are no longer triggering the enqueue behavior.
I am working on a local branch if you need any details let me know and I can post them.
On a side note I know you have code to check "delayed job" queue before adding a job twice but not for Resque queue. You might want to do something similar if you need to update the ts-resque-delta gem to a new version. Here is my code for overriding the enqueue method in order to keep the delta indexing to a minimum.
## This repalced the regular enqueue method in resque and lets us not add duplicate delta indexing
module ::Resque
def self.enqueue(klass, *args)
queue_name = queue_from_class(klass)
last_two_jobs = peek(queue_name, -2, 2) # Only check the last two queued jobs
last_two_jobs.each do |job|
# Job class matches and first argument matches
if job['class'] == klass.to_s && job["args"][0] == args[0]
# FlagAsDeletedJob has an extra id argument that must match as well
if 'FlyingSphinx::ResqueDelta::FlagAsDeletedJob' != klass.to_s || job["args"][1] == args[1]
return
end
end
end
puts "Enquing a job #{klass} args: #{args}"
enqueue_to(queue_name, klass, *args)
#.... other trigger code
end
end
You could do the above check before calling regular enqueue method. This way it still allows others to overwrite the enqueue method as I do above. I have other triggers in that method such as hiring a Heroku worker if needed. It also simplifies the code as you can do it for the particular job directly instead of both delta and deleted job. Thank you for all the work on TS3 and Flying Sphinx
Kind regards,
Conor
#1 Posted by Pat Allan Support staff on 03 Mar, 2013 03:52 AM
Hi Conor
You're right, the ts-resque-gem doesn't yet support TS v3. I do want to fix that, and maybe pull all the common logic out from ts-delayed-delta, ts-resque-delta and ts-sidekiq-delta to keep them as simple as possible.
Are you using Resque for anything else beyond the deltas? As ts-sidekiq-delta definitely works with TS v3 (we're using it for a busy-ish site without issues). I've also written a middleware for Sidekiq to have that unique checking in place: https://gist.github.com/pat/4982428
But it'd be nice to have this behaviour sorted for Resque too. Will try to get to it soon!
Cheers
Pat
#2 Posted by jack on 03 Mar, 2013 08:52 AM
Hi Pat,
Thank you for the update. Nobody told me about Sidekiq, looks much better than Resque. I use Resque for a couple of cleanup code, but easily updates. Especially since Sidekiq has no forking and uses a single connection for many workers is a feature I was missing from Resque (since I am using the mini Redis plan that limits to 10 connections). It also has middleware hooks for me to start a Heroku worker on delta enqueue as well as serialization priority (they way I stop a Heroku worker when not needed).
I'll test it tomorrow and let you know. I will use this to inject code for worker launch:
Procfile update to define priority for shutdown:
bundle exec sidekiq -q deltas -q default -q terminateI think ts-sidekiq-delta was the answer I was seeking.
I will be using your branch and not Danny's as I assume is more up to date?
Cheers,
Conor
#3 Posted by Pat Allan Support staff on 03 Mar, 2013 09:00 AM
Good to know. It's worth noting that it's my fork that has the changes, but ts-sidekiq-delta was originally forked from ts-resque-delta, so my fork of ts-sidekiq-delta ended up with the original project name:
gem 'ts-sidekiq-delta',
:git => 'git://github.com/pat/ts-resque-delta',
:branch => 'sidekiq',
:ref => '5bae80758c'
Good luck with switching everything over.
#4 Posted by jack on 03 Mar, 2013 09:05 AM
Thank you for the clarification.
#5 Posted by jack on 06 Mar, 2013 03:58 PM
Hi Pat,
Got Sidekiq up and running with TS3 and doing the delta index. Thank you for pointing me in the right direction with Sidekiq. However, I am still getting a number of duplicate jobs enqueued when updating an active record. Not sure why your unique job middleware is not being called as I see the unique option flag set from TS3..
Just wanted to give you a heads up and if I can track down the issue I will let you know. For now I updated my own middleware with a version of the above function to only add unique jobs:
On maybe a related note I get
Index Request is still pending - something has gone wrong.randomly when the delta jobs are processed by Sidekiq, even though I have Sidekiq set to serialize the jobs and do them one at a time. I'll keep an eye out and let you know if it happens often now that I am done updating to Sidekiq.#6 Posted by Pat Allan Support staff on 07 Mar, 2013 10:49 AM
Hmm, not quite sure what to make of that duplicate issue. Good to know you've got something in place to deal with it, at least.
I've seen a few people report the pending index request problem, but not yet sure what the source is. I'll have a new version of the flying-sphinx gem out soon, and that should help deal with the problem.
Thanks for the update!
jack closed this discussion on 07 Jan, 2014 10:33 AM.