Pat Allan on 27 Oct, 2016 04:44 PM
What’s the name of the app in question? Once I know that I’ll be able to investigate more deeply.
But also: occasionally I see this issue crop up from other customers, and it’s almost always because their database has changed (sometimes automatically by Heroku), but their Sphinx configuration file has not been updated accordingly. But re-running the fs:index rake task again should resolve that. I’m not quite sure what the ‘Rake aborted’ message is about… are you able to run it with --trace and share the full output with me?
Pat Allan on 29 Oct, 2016 10:43 PM
Thanks so much for the code access, that has certainly helped. And sorry it’s taken a little while to respond - I’m travelling for a conference and have been battling with a lack of time, good health, and good wifi. Still, have managed to make some progress…
From what I can see, this is related to a change in the SSL setup for flying-sphinx.com <http://flying-sphinx.com/> (I’d missed that in the previous debugging). MRI 1.9.2 (which you’re using) supports TLSv1.0, SSLv2 and SSLv3. Unfortunately, the new SSL configuration means that only TLSv1.0, TLSv1.1 and TLSv1.2 (not SSLv2 or SSLv3), which from my testing seem to have been introduced in MRI 1.9.3 onwards. Yes, TLSv1.0 is, it seems, supported in both versions of Ruby, but I’m not convinced net/http and openssl in MRI 1.9.2 can be configured to make the appropriate request type.
As you’re probably aware, Heroku no longer officially supports 1.9.x and 2.0.0 releases, though obviously you can still use the older versions (as you’re currently doing). Given Heroku’s stance, though, I’m not sure how much effort it’s worth me putting in to making 1.9.2 is reliably supported… but certainly, I’d given you and other customers no warning of this change (I hadn’t tested things with 1.9.x releases of MRI, so I wasn’t aware of any problems).
The obvious solution is to get your app running a more modern version of Ruby… but I’m well aware this is no small thing. For what it’s worth, I used the following steps on your code and managed to get the Rails console booting on MRI 1.9.3 (though I’ve not done any further testing than that, nor have I made any commits):
Added authors settings for each of the vendored engine gemspecs (this is perhaps related to the versions of rubygems/Bundler that I’m using)
Updated RedCloth to 4.2.9 in the Gemfile.
Removed ZenTest from the Gemfile
Added a specific version to better_errors in the Gemfile - 0.7.2 (which is the version that’s in your Gemfile.lock)
Added net-ssh 2.6.8 to your Gemfile (it’s in Gemfile.lock, but newer versions aren’t MRI 1.9-friendly)
Ran `bundle update RedCloth httparty eventmachine`
Of course, it could be quite possible that more changes are needed to have the app running smoothly on 1.9.3 - and I’m sure upgrading to a more modern Ruby is going to be even more work (I’ve gone through this pain for some of my own apps as well).
The alternate workaround is something I’ve not yet figured out: alter the HTTP calls being made so they use *just* TLSv1.0 (and not attempt SSLv2/3). Again: I’m not sure if this is possible… I’ve spent a decent amount of the last several hours hunting for how to do this, and I’m not finding any clear documentation thus far.
I can revert the SSL configuration for flying-sphinx.com <http://flying-sphinx.com/>, though this does mean an increase in hosting costs (which is not your problem! It’s just part of why I’d made the change). If upgrading to 1.9.3 or newer isn’t an option, do let me know, and I’ll hunt a bit more for an answer on how to configure HTTPS requests in a way that works for 1.9.2 - and if that doesn’t lead anywhere, then I’ll revert the SSL changes. But certainly, if upgrading Ruby in your app is an option, I’d greatly appreciate that!
Many thanks for your patience and understanding, and do let me know what the best ways forward for you are.
Hope you had nice time with your travel and enjoyed conference :-).
Thanks very much for looking in to this issue in your busy schedule during weekend. Really appreciating your help and sharing all the information about the issue.
Unfortunately at this moment we can't think about upgrading to any latest Ruby versions due to the Business plans already set. By the way I agree that is the best permanent solution if we seek for long running solution in Heroku.
Please let me know if you get any luck with the other solution. If not it would be really great if you can revert back the SSL changes done in FlyingSphinx. So that it might not affect our Business and would be a quick solution. Kindly let me know your thoughts. Once again thanks very much for your quick assistance :-)
Pat Allan on 31 Oct, 2016 02:07 PM
Thanks for being so understanding with this.
The only workaround I could figure out is changing the API URL
in the flying-sphinx gem to use HTTP rather than HTTPS (which I
tested on your staging app - those changes are now reverted). Both
your app and Flying Sphinx operate within Heroku’s cloud, so
the odds of requests being intercepted are super small, but still,
I just wasn't satisfied with having the communications be
So, I've reverted my SSL configuration for the domain, and
requests (without any changes to your app) are now working again -
though the DNS changes are still rolling out, so you may sometimes
see failures for the next 30-60 minutes. If you could confirm
everything is working for your non-staging environments, that'd be
Also, if/when you do change from MRI 1.9.2, do let me know -
I'll have to see if I can figure out how many other customers have
been affected by this, and plan whether it's possible to migrate
the SSL changes in a different manner.
Again, thank you for your patience, and if there are any other
issues, do let me know.