Flying-sphinx indexing has seriously slowed down since 17th April 05:00am UTC

antti's Avatar

antti

17 Apr, 2015 07:45 AM

Hi!

I'm not sure if this is something related to flying-sphinx servers, or our end, but we couldn't yet figure out any change we would have done in our end so I wanted to check if there could be any change in Flying-Sphinx that could cause this. Our index goes through a lot of data (about 16k docs) and it has took around 50 seconds according to our logs, but since 17th April 05:00am UTC the time has grown a lot and is often 130-170 seconds now. This long lasting indexing also seems to make our DB less responsive to other requests so the whole site slows down a lot. We didn't notice any effect like this when the indexing run nicely in the background in 50s or so durations.

We haven't changed anything in our indexing lately. This weird behavior seemed to start this morning and has been continuing now. Some indexes are bit faster, almost like it used to be, but most are 2 or 3 times slower than before 05am UTC today.

I see that the flying-sphinx status is on green, but let me know if there's anything you could see from your end, that could help us figure out what causes this?

Thanks!

  1. Support Staff 1 Posted by Pat Allan on 17 Apr, 2015 02:42 PM

    Pat Allan's Avatar

    Hi Antti

    There was a higher load on your server (Vulcan) for about 24 hours until 04:00 UTC today... it's now back within normal levels, and certainly this amount of load is not too common (especially for such a length of time).

    I presume this could have been part of what was making things slower - this is your production app that was having the issues, right? The last several index runs took about 20-25 seconds going by the Sphinx logs, so I think this is resolved. If you're seeing data that suggests otherwise, do let me know.

    Certainly if this load issue returns I'll be looking into it in depth - it's definitely not ideal, especially given you're paying for something beyond the entry-level plan. I am a little surprised of the impact that this would have on your database though - if Sphinx is a little slower at querying data from the DB, surely that would be less load? But if the timings of slower behaviour in your app matches the Sphinx indexing calls, then they certainly seem related.

    Kind regards,

    Pat

  2. 2 Posted by antti on 21 Apr, 2015 03:15 PM

    antti's Avatar

    Hi Pat!

    Thanks for this information. It seems that indeed our strangely long index times returned back to normal after the period you mentioned. We haven’t yet figured out why Sphinx server slowness had such big effect on our service. (probably something we could do better) but now it seems things are back to normal, so all ok for now.

    Thanks for quick response again! 

    -Antti

    —— 

    Best regards from Helsinki,

    Antti Virolainen / Co-Founder and COO at Sharetribe.

    Get in touch at [email blocked] or Skype to “gnomet".

    Create a marketplace at Sharetribe.com. It only takes a minute.

    On perjantaina, huhtikuuta 17, 2015 at 5:43 ip., Pat Allan <[email blocked]>, wrote:

  3. Support Staff 3 Posted by Pat Allan on 22 Apr, 2015 12:16 AM

    Pat Allan's Avatar

    Great to know it’s behaving as expected now :)

    I did see another burst of high CPU on Vulcan, which prompted a change to disable a CPU-intensive feature for apps using recent versions of the flying-sphinx gem (the feature is related to working around a limitation of Heroku that no longer exists, hence not needed in most cases). It was a small thing, but seems to have helped performance a bit too.

    Certainly, if anything else crops up, do let me know.

    Cheers

    Pat

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac