Heroku error when deploying thinking-sphinx gem >3.0.4: "No such file or directory - /tmp/build_xxx/log"

kevinmtrowbridge's Avatar

kevinmtrowbridge

14 Jan, 2014 10:42 PM

Hi,

I have encountered this error twice now: first when trying to upgrade to 3.0.6 ... got the error, so went back to 3.0.4, problem solved. Then today I saw there is a new 3.1.0 version of thinking-sphinx gem, so I figured the issue would probably be fixed. I tried again, but I get the same error:

No such file or directory - /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/log
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `realpath'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `realpath'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `log_root'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:114:in `configure_searchd_log_files'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:104:in `configure_searchd'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:147:in `setup'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:13:in `initialize'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:17:in `new'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:17:in `instance'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-1.0.0/lib/flying_sphinx/railtie.rb:7:in `block in <class:Railtie>'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `instance_exec'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `run'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:55:in `block in run_initializers'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `each'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `run_initializers'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:136:in `initialize!'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in `method_missing'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/config/environment.rb:5:in `<top (required)>'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `require'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `block in require'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:236:in `load_dependency'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `require'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:103:in `require_environment!'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:297:in `block (2 levels) in initialize_tasks'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:93:in `block (2 levels) in <top (required)>'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:60:in `block (3 levels) in <top (required)>'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:23:in `invoke_or_reboot_rake_task'
   /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:29:in `block (2 levels) in <top (required)>

Can you shed any light on this error?

Thanks,
Kevin

  1. 1 Posted by kevinmtrowbridg... on 14 Jan, 2014 10:46 PM

    kevinmtrowbridge's Avatar

    I see this in the 3.0.6 release notes:
    "Insist on the log directory existing, to ensure correct behaviour for symlinked paths. (Michael Pearson)."

    ... my git repository does include an empty log directory (all files are ignored, but the directory has a .gitkeep file).

  2. Support Staff 2 Posted by Pat Allan on 14 Jan, 2014 10:48 PM

    Pat Allan's Avatar

    Hi Kevin

    Thinking Sphinx is expecting a log directory to exist, and it seems your app doesn't have one (at least in the git repo). The easiest solution is to add a .git_keep file to your repo:

    touch log/.git_keep
    git add log/.git_keep
    git commit -m "Ensure the log directory exists."
    

    Unrelated to this issue, but it's worth noting that TS 3.1.0 expects Sphinx 2.1.2 or newer by default. If you don't wish to upgrade Sphinx versions, please review the release notes and make the corresponding changes:
    https://github.com/pat/thinking-sphinx/releases/tag/v3.1.0

    Flying Sphinx servers have Sphinx 2.1.4 available, and also 2.0.9 if you want the latest 2.0.x release. To switch versions, set version in the corresponding environment of config/thinking_sphinx.yml to the appropriate version number.

  3. Support Staff 3 Posted by Pat Allan on 14 Jan, 2014 10:49 PM

    Pat Allan's Avatar

    Oh, it exists? That's odd. And that .gitkeep file is definitely committed to the repo?

  4. 4 Posted by kevinmtrowbridg... on 14 Jan, 2014 11:05 PM

    kevinmtrowbridge's Avatar

    It seems to definitely exist. I can see it in Github's version of my repo. However, if I 'heroku run bash' and 'ls' -- I don't see it there. Seems like if I can get the empty directory existing on Heroku that might solve the issue. Not sure why it's not there.

  5. Support Staff 5 Posted by Pat Allan on 14 Jan, 2014 11:25 PM

    Pat Allan's Avatar

    It's especially odd because I use TS v3.0.6 in Heroku apps, and the .gitkeep file is all we have in the log directory in the git repository too.

  6. Support Staff 6 Posted by Pat Allan on 14 Jan, 2014 11:35 PM

    Pat Allan's Avatar

    So, I've just run heroku run bash on the production environment for our app, and the only file in the log directory is production.log. Our staging environment just has staging.log, and neither have the .gitkeep file. Both log files are empty, mind you, and they're not in the git repo.

    But I've not spotted anything in the default buildpack that is responsible for this behaviour...

  7. Support Staff 7 Posted by Pat Allan on 15 Jan, 2014 12:22 AM

    Pat Allan's Avatar

    Just raised the issue with some of the Heroku team on Twitter, their first suggestion is perhaps the .gitkeep file is being ignored because it's listed in your .slugignore, if you have one?

  8. 8 Posted by kevinmtrowbridg... on 15 Jan, 2014 12:30 AM

    kevinmtrowbridge's Avatar

    Ok -- I appreciate your help. As of now, the main issue seems to be that I cannot get Heroku to create a log directory in this particular app.

    I ran 'heroku run bash' on another app that I own, this one had a log directory with production.log inside, the same behavior as you describe.

    In this app, there's no log directory and no matter what I do (commit files to it, totally remove all mention of it from gitignore, etc) -- cannot get the log directory to appear in the app. Strange.

  9. 9 Posted by kevinmtrowbridg... on 15 Jan, 2014 12:32 AM

    kevinmtrowbridge's Avatar

    No, I do not have a .slugignore file -- I saw that, but sadly it was one of the first things I looked into. Getting into deep water now as to why the log directory does not exist. If I clone the repo locally, all is well. I am wondering if I have some service configured (new relic?) or Logentries that might somehow be causing this behavior.

  10. 10 Posted by kevinmtrowbridg... on 15 Jan, 2014 12:34 AM

    kevinmtrowbridge's Avatar

    Thinking maybe I should push the repo to a new Heroku app and see if the behavior is different there somehow.

  11. Support Staff 11 Posted by Pat Allan on 15 Jan, 2014 12:47 AM

    Pat Allan's Avatar

    I'm not familiar with logentries, but I doubt it's NewRelic (given we have that in our app too). Pushing the repo to a fresh Heroku app could be a good next step.

  12. 12 Posted by kevinmtrowbridg... on 15 Jan, 2014 01:02 AM

    kevinmtrowbridge's Avatar

    Pat -- I submitted an Heroku support ticket. It's not trivial to recreate the app environment enough to get it to deploy successfully. I have other things to do tonight! I will let you know when we find a resolution. For now I am reverting the app to thinking-sphinx 3.0.4.

    Thanks again for your help,
    Kevin

  13. Support Staff 13 Posted by Pat Allan on 15 Jan, 2014 01:04 AM

    Pat Allan's Avatar

    No worries - hope you can get it sorted soon. If there's any way I can help further, do let me know. Shall close this ticket for the moment though.

  14. Pat Allan closed this discussion on 15 Jan, 2014 01:04 AM.

  15. kevinmtrowbridge re-opened this discussion on 24 Jan, 2014 11:00 PM

  16. 14 Posted by kevinmtrowbridg... on 24 Jan, 2014 11:00 PM

    kevinmtrowbridge's Avatar

    Hi Pat,

    I have been corresponding with the Heroku staff about this issue and I want to relay back to you what they have told me.

    Ryan B, Heroku Support:
    "This is expected behavior that the log directory is deleted prior to the build. See #2 in this article: https://devcenter.heroku.com/articles/slug-compiler#compilation
    If you need this directory, probably the easiest thing to do is to include name the directory something like _log and then prepend your Procfile entry with mv _log log && so the directory is there at runtime. If you need this directory at build time, your other option would be to fork the Ruby buildpack and do something similar to re-create the directory before the build takes place."

    Kevin T:
    "Hi Ryan, I am not sure that is true -- the article says:
    "Remove unused files, including .git directories, .gitmodules files, anything in log and tmp, and anything specified in a top-level .slugignore file." "anything in log" -- it does not say remove the log directory itself. Both Pat Allen & myself find that the /log directory is present in other apps. The thinking-sphinx gem is dependent on the log directory being present, and having a log/.gitkeep file seems to be sufficient to keep the log directory around in other cases. The reason thinking-sphinx 3.1.0 is crashing on my instance is a change is made which expects the /log directory to be present. The reason I press the issue is that I am going to take this back to Pat Allen, which will possibly influence how he develops the thinking-sphinx gem going forward. Flying-sphinx is an addon supported by Heroku.
    Thanks,
    Kevin"

    Ryan B:
    "I just checked the code, and the directory really is deleted in its entirety. The article should be updated. If you are seeing the /log directory on other apps, there must something re-creating it either at build time or runtime. It is entirely possible that the apps you are referring to are using some buildpack that re-creates the directory because the pruning takes place between the code being unpacked from the repo and execution of the buildpack."

  17. 15 Posted by kevinmtrowbridg... on 24 Jan, 2014 11:03 PM

    kevinmtrowbridge's Avatar

    So -- if what Ryan B says is true, that change to Thinking Sphinx that throws an exception if the log directory is absent, may cause problems for other people on Heroku as well.

    For now, I will experiment with adding the lines to our Procfile to create an empty log directory.

    For a longer term solution, do you think I should create a Github issue against the TS gem to document this, or potentially investigate creating a patch that could be worked into a future release?

    Thanks,
    Kevin

  18. 16 Posted by kevinmtrowbridg... on 24 Jan, 2014 11:06 PM

    kevinmtrowbridge's Avatar

    In all honesty I don't agree with the behavior on Heroku's part to actually delete the 'log' directory. It seems if it's in git, and has a .gitkeep file, it should be kept, and deleting it will cause problems for more than just the thinking_sphinx gem.

    I don't understand why they need to be so aggressive, but then again, I don't know what problems they're trying to solve by doing this. I don't know if you have some other avenues of reaching out to them to see if this behavior could be changed.

  19. 17 Posted by kevinmtrowbridg... on 24 Jan, 2014 11:34 PM

    kevinmtrowbridge's Avatar

    Yeah -- the solution that Ryan B gave doesn't work. I still get the same crash. The command in the Procfile is not executed soon enough or somehow it's not working out:

    web: mv _log log && bundle exec unicorn -p $PORT -c ./config/unicorn.rb

    ... I don't have a solution yet for how we can upgrade past TS 3.0.4.

    The problem I'm trying to solve that set this all off, is that, on 3.0.4, we get these random exceptions in our development builds: "Encoding::UndefinedConversionError -- U+2019 from UTF-8 to ISO-8859-1"

    So for now, I am keeping the upgraded 3.1.0 running on my development build, and just avoiding checking the changes in by keeping them in a "Don't commit" changelist.

    This works for me, but I have some junior developers on my team, and they keep running into this exception. I would like to upgrade the whole app and solve the problem, but as I have been mentioning, I cannot get Heroku to take the upgraded version.

  20. Support Staff 18 Posted by Pat Allan on 26 Jan, 2014 03:56 AM

    Pat Allan's Avatar

    Hi Kevin

    I’ll try tweak TS’s behaviour so it doesn’t require a log directory. Odd that Heroku’s behaviour seems so inconsistent, but if they’re not going to change, I guess TS better change instead.

    As for the encoding error - what version of Sphinx are you using locally?


    Pat

  21. Support Staff 19 Posted by Pat Allan on 27 Jan, 2014 04:24 AM

    Pat Allan's Avatar

    Hi Kevin

    I've just pushed a fix to this log issue into the develop branch - which you can use via the following in your Gemfile:

    gem 'thinking-sphinx', '~> 3.1.0',
      :git    => 'git://github.com/pat/thinking-sphinx.git',
      :branch => 'develop',
      :ref    => 'c82ccdbe12'
    

    It's worth noting that as of the latest commit, TS now defaults to Sphinx 2.1.4 on Flying Sphinx if you're not specifying a version manually in config/thinking_sphinx.yml.

    Let me know if there's any issues.

    Pat

  22. 20 Posted by kevinmtrowbridg... on 28 Jan, 2014 09:50 PM

    kevinmtrowbridge's Avatar

    Hi Pat,

    Thank you very much for doing this. I deployed this version of the gem to our staging server, and it worked perfectly.

    I will probably wait to deploy the change into production, until it gets wrapped into an official numbered release. It's not an urgent problem, just something that I wanted to see we had resolved, because we like Flying Sphinx and want to keep up with your releases.

    Thanks again,
    Kevin

  23. Support Staff 21 Posted by Pat Allan on 24 Apr, 2014 04:08 AM

    Pat Allan's Avatar

    Hi Kevin

    Just further to this - TS v3.1.1 was released a few days ago, so you can now upgrade to a proper gem release if you wish.

    Cheers

    Pat

  24. Pat Allan closed this discussion on 24 Apr, 2014 04:08 AM.

  25. kevinmtrowbridge re-opened this discussion on 24 Apr, 2014 04:49 AM

  26. 22 Posted by kevinmtrowbridg... on 24 Apr, 2014 04:49 AM

    kevinmtrowbridge's Avatar

    Thank you Pat -- I noticed that and I've updated it in our Gemfile!

    Appreciate your help with this. Hope you are well,
    Kevin

  27. Pat Allan closed this discussion on 29 May, 2014 12:55 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac