Stopped Indexing from Sept 30 2016

Siju Thomas's Avatar

Siju Thomas

27 Oct, 2016 03:40 PM

Hi,

We have come across with an issue that any new content we add is not coming through the public or inside search. Later we just noticed that Flying Sphinx shows indexing up to Sept 30th 2016. Could you please help us to resolve this issue. Really appreciating support. Thanks much in advance.

Thank You,
Siju Thomas.

  1. 1 Posted by Siju Thomas on 27 Oct, 2016 03:57 PM

    Siju Thomas's Avatar

    We have tried to re-index using the command but failed with a message that "Rake Aborted"

  2. Support Staff 2 Posted by Pat Allan on 27 Oct, 2016 04:44 PM

    Pat Allan's Avatar

    Hi Siju

    What’s the name of the app in question? Once I know that I’ll be able to investigate more deeply.

    But also: occasionally I see this issue crop up from other customers, and it’s almost always because their database has changed (sometimes automatically by Heroku), but their Sphinx configuration file has not been updated accordingly. But re-running the fs:index rake task again should resolve that. I’m not quite sure what the ‘Rake aborted’ message is about… are you able to run it with --trace and share the full output with me?

    Thanks,


    Pat

  3. 3 Posted by mummer on 28 Oct, 2016 11:44 AM

    mummer's Avatar

    Hi Pat,

    Thanks much for your response. Our app name is "authorlearningcenter" in production and "authorlearningcenter-staging" for staging.

    We have even tried the fs:index rake but no luck. There was no deployment happened after Sept 30 from our side. What could be the reason any idea?

    Much appreciating your help.

    Thank You,
    Siju Thomas.

  4. Support Staff 4 Posted by Pat Allan on 28 Oct, 2016 02:56 PM

    Pat Allan's Avatar

    Hmm. There hasn’t been any changes to the Flying Sphinx API since August (and that was just a Rails security patch)…

    Is it possible for you to add me to the staging app ([email blocked] <mailto:[email blocked]>) so I can try debugging directly?

  5. 5 Posted by Siju Thomas on 28 Oct, 2016 03:44 PM

    Siju Thomas's Avatar

    Hi Pat,

    We have added you in the app "authorlearningcenter-staging". Please let me know your findings. Thanks much for your support.

    Thank You,
    Siju Thomas.

  6. Support Staff 6 Posted by Pat Allan on 29 Oct, 2016 10:43 PM

    Pat Allan's Avatar

    Hi Siju,

    Thanks so much for the code access, that has certainly helped. And sorry it’s taken a little while to respond - I’m travelling for a conference and have been battling with a lack of time, good health, and good wifi. Still, have managed to make some progress…

    From what I can see, this is related to a change in the SSL setup for flying-sphinx.com <http://flying-sphinx.com/> (I’d missed that in the previous debugging). MRI 1.9.2 (which you’re using) supports TLSv1.0, SSLv2 and SSLv3. Unfortunately, the new SSL configuration means that only TLSv1.0, TLSv1.1 and TLSv1.2 (not SSLv2 or SSLv3), which from my testing seem to have been introduced in MRI 1.9.3 onwards. Yes, TLSv1.0 is, it seems, supported in both versions of Ruby, but I’m not convinced net/http and openssl in MRI 1.9.2 can be configured to make the appropriate request type.

    As you’re probably aware, Heroku no longer officially supports 1.9.x and 2.0.0 releases, though obviously you can still use the older versions (as you’re currently doing). Given Heroku’s stance, though, I’m not sure how much effort it’s worth me putting in to making 1.9.2 is reliably supported… but certainly, I’d given you and other customers no warning of this change (I hadn’t tested things with 1.9.x releases of MRI, so I wasn’t aware of any problems).

    The obvious solution is to get your app running a more modern version of Ruby… but I’m well aware this is no small thing. For what it’s worth, I used the following steps on your code and managed to get the Rails console booting on MRI 1.9.3 (though I’ve not done any further testing than that, nor have I made any commits):

    Added authors settings for each of the vendored engine gemspecs (this is perhaps related to the versions of rubygems/Bundler that I’m using)
    Updated RedCloth to 4.2.9 in the Gemfile.
    Removed ZenTest from the Gemfile
    Added a specific version to better_errors in the Gemfile - 0.7.2 (which is the version that’s in your Gemfile.lock)
    Added net-ssh 2.6.8 to your Gemfile (it’s in Gemfile.lock, but newer versions aren’t MRI 1.9-friendly)
    Updated Rubygems
    Ran `bundle update RedCloth httparty eventmachine`

    Of course, it could be quite possible that more changes are needed to have the app running smoothly on 1.9.3 - and I’m sure upgrading to a more modern Ruby is going to be even more work (I’ve gone through this pain for some of my own apps as well).

    The alternate workaround is something I’ve not yet figured out: alter the HTTP calls being made so they use *just* TLSv1.0 (and not attempt SSLv2/3). Again: I’m not sure if this is possible… I’ve spent a decent amount of the last several hours hunting for how to do this, and I’m not finding any clear documentation thus far.

    I can revert the SSL configuration for flying-sphinx.com <http://flying-sphinx.com/>, though this does mean an increase in hosting costs (which is not your problem! It’s just part of why I’d made the change). If upgrading to 1.9.3 or newer isn’t an option, do let me know, and I’ll hunt a bit more for an answer on how to configure HTTPS requests in a way that works for 1.9.2 - and if that doesn’t lead anywhere, then I’ll revert the SSL changes. But certainly, if upgrading Ruby in your app is an option, I’d greatly appreciate that!

    Many thanks for your patience and understanding, and do let me know what the best ways forward for you are.

    Regards,

    Pat

  7. 7 Posted by Siju Thomas on 31 Oct, 2016 06:53 AM

    Siju Thomas's Avatar

    Hi Pat,

    Hope you had nice time with your travel and enjoyed conference :-).

    Thanks very much for looking in to this issue in your busy schedule during weekend. Really appreciating your help and sharing all the information about the issue.

    Unfortunately at this moment we can't think about upgrading to any latest Ruby versions due to the Business plans already set. By the way I agree that is the best permanent solution if we seek for long running solution in Heroku.

    Please let me know if you get any luck with the other solution. If not it would be really great if you can revert back the SSL changes done in FlyingSphinx. So that it might not affect our Business and would be a quick solution. Kindly let me know your thoughts. Once again thanks very much for your quick assistance :-)

    Thank You,
    Siju Thomas.

  8. Support Staff 8 Posted by Pat Allan on 31 Oct, 2016 02:07 PM

    Pat Allan's Avatar

    Hi Siju

    Thanks for being so understanding with this.

    The only workaround I could figure out is changing the API URL in the flying-sphinx gem to use HTTP rather than HTTPS (which I tested on your staging app - those changes are now reverted). Both your app and Flying Sphinx operate within Heroku’s cloud, so the odds of requests being intercepted are super small, but still, I just wasn't satisfied with having the communications be insecure.

    So, I've reverted my SSL configuration for the domain, and requests (without any changes to your app) are now working again - though the DNS changes are still rolling out, so you may sometimes see failures for the next 30-60 minutes. If you could confirm everything is working for your non-staging environments, that'd be great!

    Also, if/when you do change from MRI 1.9.2, do let me know - I'll have to see if I can figure out how many other customers have been affected by this, and plan whether it's possible to migrate the SSL changes in a different manner.

    Again, thank you for your patience, and if there are any other issues, do let me know.

    Pat

  9. 9 Posted by Siju Thomas on 01 Nov, 2016 12:45 PM

    Siju Thomas's Avatar

    Pat,

    Thank You Very Much for resolving the issue. We have checked and its working perfect now. Appreciating your wonderful help and support.

    Sorry for my delay in response as I was waiting for a confirmation from team that everything is back to normal.

    It was so nice talking with you and I wish to have further communication in future so that we will keep in touch :-)

    Wishing you all the best and once again thank you for all your quick responses and help.

    Thank You,
    Siju Thomas.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac