Use flying sphinx for heroku ci

Tudor Cazacu's Avatar

Tudor Cazacu

31 Mar, 2020 05:26 PM

Hello ! I'm trying to use flying sphinx add-on for setting up heroku ci.

app.json

"environments": { "test": { "env": { "POSTGRES_VERSION": "10", "REDIS_VERSION": "5" }, "addons": ["heroku-postgresql:in-dyno", "heroku-redis:in-dyno", "flying_sphinx:wooden"], "scripts": { "test-setup": "bundle exec rake test_setup", "test": "bundle exec rails test" } } }

Version:
gem 'thinking-sphinx', '~> 4.4.1'
gem 'flying-sphinx', '~> 2.1.4'

The issue is that when i try to run tests on heroku ci, i get "ThinkingSphinx::SphinxError: index event_core: index not available."
The ts:index task was run but don't seems to be effective.
I also used VERBOSE_LOGGING env var and seems all the connections are successful.

Do you have any advice ?

Thank you, Tudor

  1. Support Staff 1 Posted by Pat Allan on 02 Apr, 2020 01:14 AM

    Pat Allan's Avatar

    Hi Tudor,

    I’m not quite sure what the problem is here… I don’t suppose it’s possible for you to share the code for the test (and any test-related setup for running indexing?). Are you running the `ts:index` rake task from within your tests, or taking some other approach for indexing?


    Pat

  2. 2 Posted by Tudor Cazacu on 03 Apr, 2020 01:44 PM

    Tudor Cazacu's Avatar

    Hi Pat,

    The indexing is happen on 'test-setup' step from app.json file.
    I figured out how to do the configuration, it's working smooth now. Thank
    you !

    Best,
    Tudor

    În joi, 2 apr. 2020 la 04:14, Pat Allan <[email blocked]> a
    scris:

  3. Support Staff 3 Posted by Pat Allan on 03 Apr, 2020 01:46 PM

    Pat Allan's Avatar

    Ah, great to hear it’s working! :) If you do hit any other issues, please do let me know.

    Cheers,


    Pat

  4. 4 Posted by nexusrpg on 20 Apr, 2020 08:55 PM

    nexusrpg's Avatar

    Any help you can provide on the configuration file needed for Heroku CI? I've tried to generate one with rake ts:configure in my test-setup script and it says it generates it, but can't connect to it using that file. In my text addons section I have "flying_sphinx:wooden" listed and in test setup I have "scripts": {
            "test-setup": "bundle exec rake ts:rebuild",
            "test": "bundle exec rspec"
          }

    But I always get this at the end of test setup on HerokuCI:

    searchd is not currently running.
    Generating configuration to /app/config/test.sphinx.conf
    sh: 1: searchd: not found
    The Sphinx start command failed:
      Command: searchd --pidfile --config "/app/config/test.sphinx.conf"
      Status: 127
      Output: See above
    There may be more information about the failure in /app/log/test.searchd.log.

    I am trying to change to realtime indexes, which requires me to add callbacks to the models, so I have to have TS working to run tests with those models or the callbacks fail. Is this just a matter of the test.sphinx.conf being generated incorrectly? I'd appreciate any help here.

  5. Support Staff 5 Posted by Pat Allan on 21 Apr, 2020 01:35 AM

    Pat Allan's Avatar

    Carl, it sounds like your app is expecting Sphinx to be installed - so, it's not properly detecting Flying Sphinx as an add-on.

    I'm not sure if this is part of the problem, but: which versions of the flying-sphinx and thinking-sphinx gems are you using?

  6. 6 Posted by nexusrpg on 21 Apr, 2020 02:16 AM

    nexusrpg's Avatar

    thinking-sphinx (4.4.1)
    flying-sphinx (2.1.4)

    This is in an app that is currently using SQL indexes successfully in Production with FS wooden reindexing every 10 mins. The tests run successfully for me with Sphinx installed and running on my mac (they failed in the same way before I got it running locally). So I'm assuming this issue is either specific to Heroku CI or perhaps the testing environment?

  7. Support Staff 7 Posted by Pat Allan on 21 Apr, 2020 02:21 AM

    Pat Allan's Avatar

    I'm not particularly familiar with Heroku CI, but just to confirm: is the test-setup command run after the specified add-ons are installed? (You'll definitely want the add-ons installed before anything that works with Sphinx). And can you confirm the add-ons are indeed being installed successfully?

  8. 8 Posted by nexusrpg on 21 Apr, 2020 02:34 AM

    nexusrpg's Avatar

    That's a good question. I can see Postresql being installed at the beginning of test set up. I don't see anything about Flying Sphinx, but those are the only 2 addons I specify for test apps, so I'm not sure if it should be mentioning it at the same time or if not seeing it listed is normal. My addons section in test currently looks like this:

    "addons":[
            "heroku-postgresql:in-dyno",
            "flying_sphinx:wooden"
          ]

    Tests pass as long as they don't touch any models with real time indexes.

  9. Support Staff 9 Posted by Pat Allan on 21 Apr, 2020 03:03 AM

    Pat Allan's Avatar

    If you’re able to confirm whether the FLYING_SPHINX_IDENTIFIER environment variable is being set, that should provide a clear indication of whether the add-on is available or not - the presence of that variable is how the flying-sphinx gem knows to send commands to the API rather than to local Sphinx binaries.

  10. 10 Posted by nexusrpg on 21 Apr, 2020 03:59 AM

    nexusrpg's Avatar

    I can confirm that is set by the time the rebuild is run. It looks like a short hexadecimal id of some kind?

  11. Support Staff 11 Posted by Pat Allan on 21 Apr, 2020 04:22 AM

    Pat Allan's Avatar

    That's correct.

    Next idea as a possible cause of the problem: is the flying-sphinx gem available for all environments in your Gemfile? Especially, is it available in the test environment?

  12. 12 Posted by nexusrpg on 21 Apr, 2020 02:48 PM

    nexusrpg's Avatar

    I can't believe I forgot to check that. :face_palm: You'd think I hadn't been doing Rails development since before 1.0.

    I still have one spec that's failing and while it must be due to the spec since it has failed in the same way 4 times in a row, I don't understand why. It looks like it is likely the first index model that's tested fails with this error, but after that they all pass. Is there something I need to do to "prime" the connection before I get to an indexed model:

    # --- Caused by: ---
         # Mysql2::Error:
         # Lost connection to MySQL server at 'reading authorization packet', system error: 2

    This is happening during the setup for a test that creates a model, and after changing the test slightly it happens now during the first creation of a different instance of that model, but again, further creations of that same model pass. I'm willing to bet if I commented out this entire spec it would happen during the first time I hit a different model that is indexed (which is currently passing). But why would the first time I hit a real time indexed model fail with that authorization error, but all further tests pass?

  13. Support Staff 13 Posted by Pat Allan on 22 Apr, 2020 10:53 AM

    Pat Allan's Avatar

    You're completely right about priming the connection - essentially, the proxy that authenticates requests won't have your new build's credentials at the very beginning. I would have hoped that the auto-retrying of requests within Thinking Sphinx would have handled this, but perhaps the proxy isn't quite fast enough at caching the new details.

    So, perhaps run the following as part of your test suite loading up would help?

    begin
      ThinkingSphinx.search(populate: true)
    rescue ThinkingSphinx::Error
      # prime that Flying Sphinx proxy
    end
    
  14. 14 Posted by nexusrpg on 22 Apr, 2020 02:12 PM

    nexusrpg's Avatar

    That probably would have worked. I did something similar with attempting to create an index instance and that fixed the problem, too. However, we ran into other issues and have decided to revert the realtime indexing. It would have been nice to have, but isn't that important to us at this time. Thanks for your help on this, though.

  15. Support Staff 15 Posted by Pat Allan on 24 Apr, 2020 01:18 PM

    Pat Allan's Avatar

    Glad to know you got the suite working, but it's a shame real-time indices didn't work out. Certainly, feel free to ask further questions about that here (perhaps in a new thread) or via Thinking Sphinx's GitHub issues.

  16. Pat Allan closed this discussion on 24 Apr, 2020 01:18 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac