We need urgent help with SOLR indexing SPEEDS:
We are indexing MYSQL tables to SOLR. We are not indexing documents, we are indexing rows only and many are only IDs / Numbers. The data size is small, but rows are almost 80 BILLION total.
We have tried 3 separate ways. I need urgent help with someone who has done this type of thing before and got very fast speeds.
We are currently manage 50k rows per second (from tables in mysql with no JOINS) (using DIH)
and 15k rows per second (from tables in mysql WITH JOINS). (using DIH)
50k CONSTANT we can just accept. but JOINS we need much better (40k per sec)
Currently we need 80 BILLION rows indexing over 25 days. We have 1 large server ONLY:
192gb ram, SSD drives, 2x hexacore CPU.
Please contact me urgent if you have solution and can help.