tag:support.flying-sphinx.com,2011-01-05:/discussions/problems/2019-rebuild-not-working-for-heroku-appFlying Sphinx: Discussion 2015-10-26T16:32:56Ztag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-14T13:07:32Z2015-10-14T13:19:36ZRebuild not working for heroku app<div><p>Hi Nick</p>
<p>Can you let me know which app this is?</p>
<p>Kind regards,</p>
<p>Pat</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-14T13:17:09Z2015-10-14T13:19:36ZRebuild not working for heroku app<div><p>Hi Pat,</p>
<p>The app is "phoenix-production". Interestingly, the daemon
was<br>
automatically restarted by something - maybe a scheduled index task
- and<br>
now it appears to be working fine.</p>
<p>Thanks,<br>
Nick Branstator</p></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-14T13:20:46Z2015-10-14T13:20:46ZRebuild not working for heroku app<div><p>Thanks Nick - I thought it might be that app. I did see a glitch
and got the daemon running as quickly as I could. Yet to figure out
the cause, but I'll investigate soon. Normally monit will sort
things out itself, not sure what caused this issue just yet.</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-14T13:26:21Z2015-10-14T13:26:22ZRebuild not working for heroku app<div><p>Understood. Thanks for taking care of the restart, and will be
interested<br>
to hear what you learn from your investigation.</p>
<ul>
<li>Nick</li>
</ul></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T14:37:57Z2015-10-19T14:37:58ZRebuild not working for heroku app<div><p>Hi Pat,</p>
<p>Seems like we are still having problems: Attempts to index are
failing, and so is the rebuild action. We think index attempts may
have been failing since last Wednesday, when we previously surfaced
the problem. All of our index attempts have empty logs.</p>
<ul>
<li>Nick</li>
</ul></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T14:50:30Z2015-10-19T14:50:30ZRebuild not working for heroku app<div><p>Hi Nick</p>
<p>I just ran a full index manually, and it worked (daemon is
running as well). Trying to figure out where things are going wrong
between the Flying Sphinx API and the Sphinx indexer commands.</p>
<p>Perhaps unrelated, but just in case: can you let me know which
versions of thinking-sphinx, flying-sphinx and Ruby you're
using?</p>
<p>Will let you know when I've got something more concrete. It's my
top priority right now.</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T15:03:46Z2015-10-19T15:03:46ZRebuild not working for heroku app<div><p>Ruby 2.1.2<br>
flying-sphinx (1.2.0)<br>
thinking-sphinx (3.1.4)</p>
<p>We noticed that, while we can see your index action in the
heroku dashboard<br>
for flying sphinx, we still see no long for it.</p></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T15:15:01Z2015-10-19T15:15:01ZRebuild not working for heroku app<div><p>That was actually a second test I made, through the API, and it
seemed to have the same problem as your calls. It seems to be
related to Kernel and/or STDIN/STDOUT not communicating correctly
from Sidekiq (on your specific Sphinx server). Restarting Sidekiq
has fixed the issue, so I guess that's a short-term fix in
place.</p>
<p>Annoyingly, there are no exceptions being raised, but I'll put
something custom in place (essentially, no indexer log = raise) so
I can at least track if/when it next happens and look at what may
have caused it in more detail.</p>
<p>For now, though, things should be operating properly. If you hit
any issues, do let me know.</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T15:15:36Z2015-10-19T15:15:36ZRebuild not working for heroku app<div><p>Additionally, the reindex that you executed does not actually
seem to have<br>
updated the index; at the least, we are not seeing any of the new
data that<br>
should be visible.</p>
<ul>
<li>Nick</li>
</ul></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T15:56:52Z2015-10-19T15:56:52ZRebuild not working for heroku app<div><p>Well, that placeholder exception's being raised more often than
not. The fact that it's inconsistent is particularly
frustrating… I'm going to keep hunting through why it's
happening.</p>
<p>The catch is I've got two flights to catch (the first is just
over an hour, the second is three and a half hours), with a break
of two hours or so in-between, so my communications and debugging
is going to be hindered a bit by that, but whenever I can be online
working on it, I will be (and in the meantime, I'll be trying to
reproduce the issue offline).</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-19T15:59:12Z2015-10-19T15:59:12ZRebuild not working for heroku app<div><p>The indexer data files are definitely being updated, and Sphinx
is rotating the new files into place, so I'd expect results to be
up-to-date. Can you run me through the data you're expecting to see
and the search queries you're running?</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-20T01:49:34Z2015-10-20T01:49:34ZRebuild not working for heroku app<div><p>Hi Nick</p>
<p>Very sorry for the delay on this. Have got through my flights,
worked through the problem, and things are working now. Redis has
been upgraded, which is helping Sidekiq run more smoothly, and I'm
not seeing any more IO errors. I'll keep an eye on things, but I've
just run several index calls (via the API, so, same behaviour as
what you should see), and the output is coming through
properly.</p>
<p>If you're still seeing data not appearing which you'd expect to
see, let's talk through the queries and the expected data and try
to debug that further.</p>
<p>Many thanks for your patience.</p>
<p>Pat</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-26T15:39:55Z2015-10-26T15:39:56ZRebuild not working for heroku app<div><p>Hi Pat,</p>
<p>We are having a problem again. We requested a simple rebuild
after a<br>
production push of our application today; the rebuild has now been
running<br>
for over two hours. Normally it takes place within about 30
minutes. The<br>
log on the dashboard has not updated in more than an hour.</p>
<ul>
<li>Nick</li>
</ul></div>Nick Branstatortag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-26T16:04:26Z2015-10-26T16:04:26ZRebuild not working for heroku app<div><p>Daemon's now back up, am debugging the issue further.</p></div>Pat Allantag:support.flying-sphinx.com,2011-01-05:Comment/381880712015-10-26T16:32:52Z2015-10-26T16:32:52ZRebuild not working for heroku app<div><p>Have put more detailed logging in place, and now the error's
disappeared - which is frustrating, but seems to be par for the
course for this issue. It's also odd that you're the only customer
it's cropping up for (thus, particularly annoying for you, and
fewer data points for me).</p>
<p>If I find anything further, I'll let you know.</p></div>Pat Allan