Heroku error when deploying thinking-sphinx gem >3.0.4: "No such file or directory - /tmp/build_xxx/log"
Hi,
I have encountered this error twice now: first when trying to upgrade to 3.0.6 ... got the error, so went back to 3.0.4, problem solved. Then today I saw there is a new 3.1.0 version of thinking-sphinx gem, so I figured the issue would probably be fixed. I tried again, but I get the same error:
No such file or directory - /tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/log
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `realpath'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `realpath'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:120:in `log_root'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:114:in `configure_searchd_log_files'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:104:in `configure_searchd'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:147:in `setup'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:13:in `initialize'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:17:in `new'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/configuration.rb:17:in `instance'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/flying-sphinx-1.0.0/lib/flying_sphinx/railtie.rb:7:in `block in <class:Railtie>'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `instance_exec'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:30:in `run'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:55:in `block in run_initializers'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `each'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/initializable.rb:54:in `run_initializers'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:136:in `initialize!'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in `method_missing'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/config/environment.rb:5:in `<top (required)>'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `require'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `block in require'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:236:in `load_dependency'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/dependencies.rb:251:in `require'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:103:in `require_environment!'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:297:in `block (2 levels) in initialize_tasks'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:93:in `block (2 levels) in <top (required)>'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:60:in `block (3 levels) in <top (required)>'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:23:in `invoke_or_reboot_rake_task'
/tmp/build_36d89422-e8c0-4b48-b1ba-8422316a92d7/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/sprockets/assets.rake:29:in `block (2 levels) in <top (required)>
Can you shed any light on this error?
Thanks,
Kevin
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
? | Show this help |
---|---|
ESC | Blurs the current field |
Comment Form
r | Focus the comment reply box |
---|---|
^ + ↩ | Submit the comment |
You can use Command ⌘
instead of Control ^
on Mac
1 Posted by kevinmtrowbridg... on Jan 14, 2014 @ 10:46 PM
I see this in the 3.0.6 release notes:
"Insist on the log directory existing, to ensure correct behaviour for symlinked paths. (Michael Pearson)."
... my git repository does include an empty log directory (all files are ignored, but the directory has a .gitkeep file).
Support Staff 2 Posted by Pat Allan on Jan 14, 2014 @ 10:48 PM
Hi Kevin
Thinking Sphinx is expecting a log directory to exist, and it seems your app doesn't have one (at least in the git repo). The easiest solution is to add a .git_keep file to your repo:
Unrelated to this issue, but it's worth noting that TS 3.1.0 expects Sphinx 2.1.2 or newer by default. If you don't wish to upgrade Sphinx versions, please review the release notes and make the corresponding changes:
https://github.com/pat/thinking-sphinx/releases/tag/v3.1.0
Flying Sphinx servers have Sphinx 2.1.4 available, and also 2.0.9 if you want the latest 2.0.x release. To switch versions, set
version
in the corresponding environment ofconfig/thinking_sphinx.yml
to the appropriate version number.Support Staff 3 Posted by Pat Allan on Jan 14, 2014 @ 10:49 PM
Oh, it exists? That's odd. And that .gitkeep file is definitely committed to the repo?
4 Posted by kevinmtrowbridg... on Jan 14, 2014 @ 11:05 PM
It seems to definitely exist. I can see it in Github's version of my repo. However, if I 'heroku run bash' and 'ls' -- I don't see it there. Seems like if I can get the empty directory existing on Heroku that might solve the issue. Not sure why it's not there.
Support Staff 5 Posted by Pat Allan on Jan 14, 2014 @ 11:25 PM
It's especially odd because I use TS v3.0.6 in Heroku apps, and the .gitkeep file is all we have in the log directory in the git repository too.
Support Staff 6 Posted by Pat Allan on Jan 14, 2014 @ 11:35 PM
So, I've just run
heroku run bash
on the production environment for our app, and the only file in the log directory is production.log. Our staging environment just has staging.log, and neither have the .gitkeep file. Both log files are empty, mind you, and they're not in the git repo.But I've not spotted anything in the default buildpack that is responsible for this behaviour...
Support Staff 7 Posted by Pat Allan on Jan 15, 2014 @ 12:22 AM
Just raised the issue with some of the Heroku team on Twitter, their first suggestion is perhaps the .gitkeep file is being ignored because it's listed in your .slugignore, if you have one?
8 Posted by kevinmtrowbridg... on Jan 15, 2014 @ 12:30 AM
Ok -- I appreciate your help. As of now, the main issue seems to be that I cannot get Heroku to create a log directory in this particular app.
I ran 'heroku run bash' on another app that I own, this one had a log directory with production.log inside, the same behavior as you describe.
In this app, there's no log directory and no matter what I do (commit files to it, totally remove all mention of it from gitignore, etc) -- cannot get the log directory to appear in the app. Strange.
9 Posted by kevinmtrowbridg... on Jan 15, 2014 @ 12:32 AM
No, I do not have a .slugignore file -- I saw that, but sadly it was one of the first things I looked into. Getting into deep water now as to why the log directory does not exist. If I clone the repo locally, all is well. I am wondering if I have some service configured (new relic?) or Logentries that might somehow be causing this behavior.
10 Posted by kevinmtrowbridg... on Jan 15, 2014 @ 12:34 AM
Thinking maybe I should push the repo to a new Heroku app and see if the behavior is different there somehow.
Support Staff 11 Posted by Pat Allan on Jan 15, 2014 @ 12:47 AM
I'm not familiar with logentries, but I doubt it's NewRelic (given we have that in our app too). Pushing the repo to a fresh Heroku app could be a good next step.
12 Posted by kevinmtrowbridg... on Jan 15, 2014 @ 01:02 AM
Pat -- I submitted an Heroku support ticket. It's not trivial to recreate the app environment enough to get it to deploy successfully. I have other things to do tonight! I will let you know when we find a resolution. For now I am reverting the app to thinking-sphinx 3.0.4.
Thanks again for your help,
Kevin
Support Staff 13 Posted by Pat Allan on Jan 15, 2014 @ 01:04 AM
No worries - hope you can get it sorted soon. If there's any way I can help further, do let me know. Shall close this ticket for the moment though.
Pat Allan closed this discussion on Jan 15, 2014 @ 01:04 AM.
kevinmtrowbridge re-opened this discussion on Jan 24, 2014 @ 11:00 PM
14 Posted by kevinmtrowbridg... on Jan 24, 2014 @ 11:00 PM
Hi Pat,
I have been corresponding with the Heroku staff about this issue and I want to relay back to you what they have told me.
Ryan B, Heroku Support:
"This is expected behavior that the log directory is deleted prior to the build. See #2 in this article: https://devcenter.heroku.com/articles/slug-compiler#compilation
If you need this directory, probably the easiest thing to do is to include name the directory something like _log and then prepend your Procfile entry with mv _log log && so the directory is there at runtime. If you need this directory at build time, your other option would be to fork the Ruby buildpack and do something similar to re-create the directory before the build takes place."
Kevin T:
"Hi Ryan, I am not sure that is true -- the article says:
"Remove unused files, including .git directories, .gitmodules files, anything in log and tmp, and anything specified in a top-level .slugignore file." "anything in log" -- it does not say remove the log directory itself. Both Pat Allen & myself find that the /log directory is present in other apps. The thinking-sphinx gem is dependent on the log directory being present, and having a log/.gitkeep file seems to be sufficient to keep the log directory around in other cases. The reason thinking-sphinx 3.1.0 is crashing on my instance is a change is made which expects the /log directory to be present. The reason I press the issue is that I am going to take this back to Pat Allen, which will possibly influence how he develops the thinking-sphinx gem going forward. Flying-sphinx is an addon supported by Heroku.
Thanks,
Kevin"
Ryan B:
"I just checked the code, and the directory really is deleted in its entirety. The article should be updated. If you are seeing the /log directory on other apps, there must something re-creating it either at build time or runtime. It is entirely possible that the apps you are referring to are using some buildpack that re-creates the directory because the pruning takes place between the code being unpacked from the repo and execution of the buildpack."
15 Posted by kevinmtrowbridg... on Jan 24, 2014 @ 11:03 PM
So -- if what Ryan B says is true, that change to Thinking Sphinx that throws an exception if the log directory is absent, may cause problems for other people on Heroku as well.
For now, I will experiment with adding the lines to our Procfile to create an empty log directory.
For a longer term solution, do you think I should create a Github issue against the TS gem to document this, or potentially investigate creating a patch that could be worked into a future release?
Thanks,
Kevin
16 Posted by kevinmtrowbridg... on Jan 24, 2014 @ 11:06 PM
In all honesty I don't agree with the behavior on Heroku's part to actually delete the 'log' directory. It seems if it's in git, and has a .gitkeep file, it should be kept, and deleting it will cause problems for more than just the thinking_sphinx gem.
I don't understand why they need to be so aggressive, but then again, I don't know what problems they're trying to solve by doing this. I don't know if you have some other avenues of reaching out to them to see if this behavior could be changed.
17 Posted by kevinmtrowbridg... on Jan 24, 2014 @ 11:34 PM
Yeah -- the solution that Ryan B gave doesn't work. I still get the same crash. The command in the Procfile is not executed soon enough or somehow it's not working out:
web: mv _log log && bundle exec unicorn -p $PORT -c ./config/unicorn.rb
... I don't have a solution yet for how we can upgrade past TS 3.0.4.
The problem I'm trying to solve that set this all off, is that, on 3.0.4, we get these random exceptions in our development builds: "Encoding::UndefinedConversionError -- U+2019 from UTF-8 to ISO-8859-1"
So for now, I am keeping the upgraded 3.1.0 running on my development build, and just avoiding checking the changes in by keeping them in a "Don't commit" changelist.
This works for me, but I have some junior developers on my team, and they keep running into this exception. I would like to upgrade the whole app and solve the problem, but as I have been mentioning, I cannot get Heroku to take the upgraded version.
Support Staff 18 Posted by Pat Allan on Jan 26, 2014 @ 03:56 AM
Hi Kevin
I’ll try tweak TS’s behaviour so it doesn’t require a log directory. Odd that Heroku’s behaviour seems so inconsistent, but if they’re not going to change, I guess TS better change instead.
As for the encoding error - what version of Sphinx are you using locally?
—
Pat
Support Staff 19 Posted by Pat Allan on Jan 27, 2014 @ 04:24 AM
Hi Kevin
I've just pushed a fix to this log issue into the develop branch - which you can use via the following in your Gemfile:
It's worth noting that as of the latest commit, TS now defaults to Sphinx 2.1.4 on Flying Sphinx if you're not specifying a version manually in config/thinking_sphinx.yml.
Let me know if there's any issues.
Pat
20 Posted by kevinmtrowbridg... on Jan 28, 2014 @ 09:50 PM
Hi Pat,
Thank you very much for doing this. I deployed this version of the gem to our staging server, and it worked perfectly.
I will probably wait to deploy the change into production, until it gets wrapped into an official numbered release. It's not an urgent problem, just something that I wanted to see we had resolved, because we like Flying Sphinx and want to keep up with your releases.
Thanks again,
Kevin
Support Staff 21 Posted by Pat Allan on Apr 24, 2014 @ 04:08 AM
Hi Kevin
Just further to this - TS v3.1.1 was released a few days ago, so you can now upgrade to a proper gem release if you wish.
Cheers
Pat
Pat Allan closed this discussion on Apr 24, 2014 @ 04:08 AM.
kevinmtrowbridge re-opened this discussion on Apr 24, 2014 @ 04:49 AM
22 Posted by kevinmtrowbridg... on Apr 24, 2014 @ 04:49 AM
Thank you Pat -- I noticed that and I've updated it in our Gemfile!
Appreciate your help with this. Hope you are well,
Kevin
Pat Allan closed this discussion on May 29, 2014 @ 12:55 PM.