solr indexing expert needed for some advice java urgent

  • Status Closed
  • Budget $250 - $750 USD
  • Total Bids 8

Project Description

We need urgent help with SOLR indexing SPEEDS:

We are indexing MYSQL tables to SOLR. We are not indexing documents, we are indexing rows only and many are only IDs / Numbers. The data size is small, but rows are almost 80 BILLION total.

We have tried 3 separate ways. I need urgent help with someone who has done this type of thing before and got very fast speeds.

We are currently manage 50k rows per second (from tables in mysql with no JOINS) (using DIH)

and 15k rows per second (from tables in mysql WITH JOINS). (using DIH)

50k CONSTANT we can just accept. but JOINS we need much better (40k per sec)

Currently we need 80 BILLION rows indexing over 25 days. We have 1 large server ONLY:

192gb ram, SSD drives, 2x hexacore CPU.

Please contact me urgent if you have solution and can help.

Thanks

Get free quotes for a project like this
Skills Required

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online