Pat Allan on 14 Oct, 2015 01:20 PM
Thanks Nick - I thought it might be that app. I did see a glitch
and got the daemon running as quickly as I could. Yet to figure out
the cause, but I'll investigate soon. Normally monit will sort
things out itself, not sure what caused this issue just yet.
on 19 Oct, 2015 02:37 PM
Seems like we are still having problems: Attempts to index are failing, and so is the rebuild action. We think index attempts may have been failing since last Wednesday, when we previously surfaced the problem. All of our index attempts have empty logs.
Pat Allan on 19 Oct, 2015 03:15 PM
That was actually a second test I made, through the API, and it
seemed to have the same problem as your calls. It seems to be
related to Kernel and/or STDIN/STDOUT not communicating correctly
from Sidekiq (on your specific Sphinx server). Restarting Sidekiq
has fixed the issue, so I guess that's a short-term fix in
Annoyingly, there are no exceptions being raised, but I'll put
something custom in place (essentially, no indexer log = raise) so
I can at least track if/when it next happens and look at what may
have caused it in more detail.
For now, though, things should be operating properly. If you hit
any issues, do let me know.
Pat Allan on 19 Oct, 2015 03:56 PM
Well, that placeholder exception's being raised more often than
not. The fact that it's inconsistent is particularly
frustrating… I'm going to keep hunting through why it's
The catch is I've got two flights to catch (the first is just
over an hour, the second is three and a half hours), with a break
of two hours or so in-between, so my communications and debugging
is going to be hindered a bit by that, but whenever I can be online
working on it, I will be (and in the meantime, I'll be trying to
reproduce the issue offline).
Pat Allan on 19 Oct, 2015 03:59 PM
The indexer data files are definitely being updated, and Sphinx
is rotating the new files into place, so I'd expect results to be
up-to-date. Can you run me through the data you're expecting to see
and the search queries you're running?
Pat Allan on 20 Oct, 2015 01:49 AM
Very sorry for the delay on this. Have got through my flights,
worked through the problem, and things are working now. Redis has
been upgraded, which is helping Sidekiq run more smoothly, and I'm
not seeing any more IO errors. I'll keep an eye on things, but I've
just run several index calls (via the API, so, same behaviour as
what you should see), and the output is coming through
If you're still seeing data not appearing which you'd expect to
see, let's talk through the queries and the expected data and try
to debug that further.
on 26 Oct, 2015 03:39 PM
We are having a problem again. We requested a simple rebuild after a
production push of our application today; the rebuild has now been running
for over two hours. Normally it takes place within about 30 minutes. The
log on the dashboard has not updated in more than an hour.
Pat Allan on 26 Oct, 2015 04:32 PM
Have put more detailed logging in place, and now the error's
disappeared - which is frustrating, but seems to be par for the
course for this issue. It's also odd that you're the only customer
it's cropping up for (thus, particularly annoying for you, and
fewer data points for me).