Hey Pat and team,
We are experiencing some data loss in production and need to rectify it ASAP. Here is our information:
Server: dusseldorf —
ec2-23-20-111-167.compute-1.amazonaws.com
Plan: Ceramic Sphinx — $55.00/month
Memory Usage: 148 MB
Disk Usage: 382 MB
Let me know if you need anything else from us.
Jordan Fowler
#1 Posted by Pat Allan Support staff on 06 Sep, 2012 10:40 AM
Hi Jordan
What's your app's name? And how is the data loss revealing itself to you? Is it records not appearing, or errors providing a bit more detail?
Cheers
Pat
#2 Posted by Jordan Fowler on 06 Sep, 2012 06:52 PM
Hey Pat,
Things are definitely wonky. I went ahead and rebuilt our index and things were find for a while. Through the Index Logs, I was able to find this:
using config file '/mnt/local/flying-sphinx/37623339543044b07/sphinx.conf'... indexing index 'record_delta'... ERROR: index 'record_delta': sql_range_query: ERROR: out of memory DETAIL: Failed on request of size 536870912. (DSN=pgsql://u7kkvti8qnhlfn:***@127.0.0.1:10341/df0smpggml78t6). total 230 docs, 120074 bytes total 361.848 sec, 331 bytes/sec, 0.63 docs/sec total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg total 17 writes, 0.269 sec, 7710.1 kb/call avg, 15.8 msec/call avg
The key is ERROR: out of memory.
Can you tell me whether t his is on you guys or on postgres?
#3 Posted by Pat Allan Support staff on 06 Sep, 2012 07:01 PM
Heya Jordan
On a train, so my debugging options are limited.
I did see that error - didn't realise it was your app. When I was googling earlier it seemed to be related to Postgres rather than Sphinx, but I'm not yet certain. The delta indices failing is the likely cause for missing data in searches.
When I have a moment later this evening, I'll try to confirm whether it's a PostgreSQL thing. Feel free to research it yourself too :)
Pat
#4 Posted by Pat Allan Support staff on 06 Sep, 2012 10:02 PM
Definitely seems like the error is something coming from PostgreSQL, so it may be worth talking to the Heroku team about this. Are you still on one of their deprecated shared databases, or have you migrated to one of the newer plans?