Missing data

Gordon's Avatar


14 Sep, 2020 09:19 PM

We have an issue where some records are not showing up in our search and I have no idea why. Would there be any error report available if a single record could not be indexed? It looks like our indexing is generally working –– logs don’t show anything unusual. Is it possible that our plan is too small and some records are being omitted?

  1. Support Staff 1 Posted by Pat Allan on 15 Sep, 2020 12:28 AM

    Pat Allan's Avatar

    Hi Gordon,

    The plan limits don’t have any impact on data being available or not in such a manner, and I’m not seeing any errors coming through either. Can you let me know what the name of your app is, and I can investigate further.

    Kind regards,


  2. 2 Posted by Gordon B. Isnor on 15 Sep, 2020 01:16 AM

    Gordon B. Isnor's Avatar

    ok thanks pat


  3. Support Staff 3 Posted by Pat Allan on 15 Sep, 2020 03:58 AM

    Pat Allan's Avatar

    Hi Gordon,

    Thanks for the app name. I’m not spotting any issues thus far… the indexing process you’re running daily is successfully processing each of the indices. Also, even though it has no impact, you’re well under the plan limits anyway.

    Are the records that aren’t coming through always the same ones? Do they appear after the indexing process has run, or is the problem persisting? Can you talk through a specific example, which model is involved, any filters, etc? I’m wondering if either the index setup or search query needs some tweaking.



  4. 4 Posted by Gordon B. Isnor on 15 Sep, 2020 02:27 PM

    Gordon B. Isnor's Avatar


       - It appears that it’s newer records that are affected
       - The problem is persistent
       - Search works for the book model up to the record with ID 757 –
       e.g. Book.search “9781773382104”
       - There’s a gap in primary keys after that, with records resuming at ID
       - All records above 790 are not available via search – e.g. Book.search

    I assume that my client deleted the records 758 - 779 for some reason (I
    can do some research and see if I can find out what that missing block is)
    –– now I’m wondering if that gap is somehow causing trouble…


  5. Support Staff 5 Posted by Pat Allan on 15 Sep, 2020 02:53 PM

    Pat Allan's Avatar

    Hi Gordon,

    First, I’ve confirmed that book records above 757 weren’t being returned, in particular 790. And then I checked the SQL queries that the indexer runs, and they should be returning that record. There are 426 books in your database, and the indexer output says 426 books are stored in Sphinx. So the fact it wasn’t there is very odd.

    Then I ran the indexer process manually - again, it says 426 books - and the record for 790 is now there (and matches by ISBN13). I don’t think I’ve done anything different to what’s already happening, so it’s really very odd. I’m happy that the result is there now, but the fact it wasn’t previously - even with daily indexing - is something I can’t yet explain.

    Also: the gap shouldn’t have had any impact at all, so I don’t think you need to worry about the deleted books.

    It’s approaching 1am here, so I’m going to sleep on this and see if any inspiration comes to me.

    One thing that’s worth doing - though I don’t expect it’s related to this issue - is switching to Sphinx 2.2.11, as that’s the latest stable release to support PostgreSQL.
    http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version <http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version>
    Once you’ve updated, you’ll want to run the rebuild task to ensure everything’s running correctly on the new version. That’ll mean a few seconds of downtime for Sphinx, so timing it with a quiet moment on the site is wise.

    Also: were you only seeing this issue with books, or with other models as well? And which versions of the thinking-sphinx and flying-sphinx gems are you using?


  6. 6 Posted by Gordon B. Isnor on 15 Sep, 2020 09:16 PM

    Gordon B. Isnor's Avatar


       - We do seem to be on the latest version of flying-sphinx and
       - I have updated to Sphinx 2.2.11
       - The only missing records I am aware of is those, but I can take a look
       at some other models to see if I can discover that anything else is going
       - Let me know if you have any ideas about how those records were getting
       lost in the shuffle


  7. Support Staff 7 Posted by Pat Allan on 16 Sep, 2020 04:49 AM

    Pat Allan's Avatar

    Great to know everything’s up-to-date.

    I’m still not sure of the original cause. If you spot any other issues, do let me know!

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac