Home → Problems →

Failure to start: No such file or directory; NOT SERVING

• Edit

Sean Massa

26 Jun, 2019 07:12 PM

I'm trying to get this set up on heroku. I have it working just fine locally.

The config loaded and the index created successfully. But when I try to start the service, it can't find my index file.

The service appears to start just fine:

listening on all interfaces, port=9313

but the "start" action fails because:

precaching index 'MyIndex'
[Wed Jun 26 19:03:32.984 2019] [ 9700] WARNING: index 'MyIndex': preload: failed to open /mnt/local/flying-sphinx/552dde8b36b080ec0/indexes/my-index.sph: No such file or directory; NOT SERVING
[Wed Jun 26 19:03:32.984 2019] [ 9700] FATAL: no valid indexes to serve
'

It looks like Flying Sphinx overwrites the paths to various files from my config file, which is fine, but it also means I don't know how to adjust for this problem. Any suggestions?

1. 1 Posted by Sean Massa on 26 Jun, 2019 07:45 PM

   

   It looks like trying again several times resolved the problem.

   • Edit

2. Support Staff 2 Posted by Pat Allan on 27 Jun, 2019 03:48 AM

   

   Hi Sean,

   Sorry that something didn’t quite go to plan the first time around - though it’s good to know things are working more smoothly now.

   I’ll definitely have a look to try to make sure such issues don’t crop up again. Can you let me know which version/language of the flying-sphinx client library you’re using?

   Thanks,

   —
   Pat

   • Edit

3. 3 Posted by Sean Massa on 03 Jul, 2019 03:05 AM

   

   Sure! I'm using Node.js libraries "flying-sphinx@0.2.3" and "sphinx-promise@1.0.5", which depends on "sphinxapi@1.3.0".

   • Edit

4. Support Staff 4 Posted by Pat Allan on 14 Jul, 2019 12:47 PM

   

   Hi Sean,

   Thanks for providing those details. The node client had been neglected for quite some time, but I’ve spent a good part of the last week updating it significantly. It now uses the v5 Flying Sphinx API (previously it’d been using v3), which is more reliable, using websockets to track action results (rather than polling, or just running long-lived requests).

   These changes are released as v1.0.0 of the NPM package - if you have the chance to give it a shot, that’d be great! Do note the updated approach for managing configuration:
   https://github.com/flying-sphinx/flying-sphinx-js#configuration <https://github.com/flying-sphinx/flying-sphinx-js#configuration>
   (But the old approach should continue to work, and none of the public syntax has changed.)

   One thing that’s now possible with the client updates and the configuration changes is the ability to specify the version of Sphinx you want to use. The old default was v2.0.6, which is quite ancient in Sphinx terms. I recommend updating to v2.2.11 (or even 3.1.1 - depending on what you use locally, of course!). Both of these versions should improve performance, and possibly reduce the Sphinx index file size. However: making this change will result in the need to rebuild your Sphinx setup (taking the daemon offline while the re-indexing happens), so definitely give this a spin on your staging environment first, and plan the production update for a time/situation where Sphinx being down will impact as few people as possible.

   If you hit any issues with this new NPM package version, do let me know. I’ve tested it out (and it has a test suite covering all current Node versions), but definitely keen for feedback and feature suggestions from customers. Ruby is the language that has historically gotten the most love by Flying Sphinx customers, hence I’m especially keen on input from those using different languages.

   Kind regards,

   —
   Pat

   • Edit

5. 5 Posted by Sean Massa on 15 Jul, 2019 01:43 PM

   

   Hey Pat! We'll check out the new version and let you know how it goes.

   Right now, we're having the same issue again. Now that we're using this in production, our production searches are down. Can you look into this for us?

   • Edit

6. 6 Posted by Sean Massa on 15 Jul, 2019 01:45 PM

   

   Oh! I just saw that we received an error email, but I'm still not sure what to do about it.

   ERROR: index 'JobIndex': raw_hits: write error: 92303 of 1048450 bytes written.

   • Edit

7. Support Staff 7 Posted by Pat Allan on 15 Jul, 2019 01:57 PM

   

   Hi Sean,

   I’m just having a look now, but it might be that your disk usage is actually jumping up significantly and you’re maxing out the server’s disk space:
   https://stackoverflow.com/questions/4367556/what-does-the-error-error-index-product-raw-hits-write-error-122070-of-26 <https://stackoverflow.com/questions/4367556/what-does-the-error-error-index-product-raw-hits-write-error-122070-of-26>

   I can see that you’re running another indexing request now, so I’m keeping an eye on the disk space. There’s about 45GB free… so using all of that would be rather worrying - but you’re already using 20GB! I’m going to kill your existing indexer process, remove all existing index files (given they’re not usable at this point), just to ensure there’s as much disk space available as possible, and then run the indexer myself.

   I’ll let you know how it goes.

   —
   Pat

   • Edit

8. Support Staff 8 Posted by Pat Allan on 15 Jul, 2019 02:03 PM

   

   Just further to this, while the indexer is running again:

   • From memory, your first full index process took over half an hour - so our best case scenario is probably that kind of timeframe - unless all the disk space disappears again!
   • Upgrading Sphinx to v2.2.11 might help speed and disk usage - given you've got this downtime anyway, should I manually set it to use v2.2.11 instead of v2.0.6? (The implications of this is that you shouldn't run the configure command using the 0.2.3 version of the flying-sphinx package, as that'd overwrite things back to v2.0.6).
   • How many records are in your jobs table? We've hit about 200k records processed thus far.
   • Edit

9. 9 Posted by Sean Massa on 15 Jul, 2019 02:12 PM

   

   Thanks for checking on this!

   Sphinx Version: Yeah, if you think that'll help. I'm updating the node package locally now anyway.

   Record Count: 1,236,233

   Disk Space: We were including a large "description" column before, but have removed that. We did add more records since the last build, but I was hoping the column removal would make up for that in disk size. Let's see where we end up after this new clean run.

   • Edit

10. Support Staff 10 Posted by Pat Allan on 15 Jul, 2019 02:17 PM

    

    Just killed my first manual attempt (using 2.0.6), switched to 2.2.11…

    The 2.0.6 run had gotten to 324k records in 16 minutes. The 2.2.11 run has just hit the same mark in 4 minutes.

    • Edit

11. 11 Posted by Sean Massa on 15 Jul, 2019 02:20 PM

    

    Oh nice! Thanks for the update!

    Does the new node package default to using Sphinx 2.2.11? That's what we use locally. I'm mostly wondering if I need to use the JavaScript configuration at all. If so, does it need to look like this?

    configuration.process('configure', function(configurer) {
        configurer.addEngine('sphinx');
    
        // http://support.flying-sphinx.com/kb/configuration/setting-your-sphinx-version
        configurer.addVersion('2.2.11');
    });
    

    I'll hold off on pushing anything, for sure. I'm just wondering for after we figure this out.

    • Edit

12. Support Staff 12 Posted by Pat Allan on 15 Jul, 2019 02:25 PM

    

    The new v1.0.0 NPM package will not overwrite the version if you don't specify one (v0.2.3 will, though).

    Granted, it doesn't hurt to have it set as part of your configuration process, but yeah, that's not something you need to do right now.

    • Edit

13. 13 Posted by Sean Massa on 15 Jul, 2019 02:39 PM

    

    I see search results again! I have a change ready on our end to upgrade the npm package and specify sphinx version 2.2.11. Should I deploy that?

    • Edit

14. Support Staff 14 Posted by Pat Allan on 15 Jul, 2019 02:40 PM

    

    May have been a brief blip with the daemon just then - it had booted up with 2.0.6 thanks to monit, but is now 2.2.11. Feel free to deploy the new NPM package :)

    • Edit

15. Support Staff 15 Posted by Pat Allan on 15 Jul, 2019 02:40 PM

    

    Disk usage is down to 1.4GB too! 🎉

    • Edit

16. Support Staff 16 Posted by Pat Allan on 15 Jul, 2019 02:42 PM

    

    The output of the indexer, given that because it was a manual invocation it won't appear in your Flying Sphinx dashboard:

    Sphinx 2.2.11-id64-release (95ae9a6)
    Copyright (c) 2001-2016, Andrew Aksyonoff
    Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
    
    using config file '/mnt/local/flying-sphinx/552dde8b36b080ec0/sphinx.conf'...
    indexing index 'JobIndex'...
    collected 1236233 docs, 3131.0 MB
    sorted 361.1 Mhits, 100.0% done
    total 1236233 docs, 3130995478 bytes
    total 1492.916 sec, 2097234 bytes/sec, 828.06 docs/sec
    total 544 reads, 0.481 sec, 2823.8 kb/call avg, 0.8 msec/call avg
    total 3046 writes, 2.066 sec, 989.7 kb/call avg, 0.6 msec/call avg
    

    Took 25 minutes.

    • Edit

17. 17 Posted by Sean Massa on 15 Jul, 2019 02:43 PM

    

    Great! Thanks a bunch for your help.

    • Edit

18. Support Staff 18 Posted by Pat Allan on 15 Jul, 2019 02:45 PM

    

    No worries at all, great to have everything upgraded and working faster and with lower disk usage. If there's any other queries or issues, do let me know. (Though right now it's approaching 1am here in Melbourne, so I'll likely be asleep for any immediate questions!)

    • Edit