Closed

SOLR large data indexing help

Hi,

We deal with huge solr data (huge rows - 10 billion, but only small size - 200GB) We are trying to index it to solr in fastest way from mysql DB.

The solr importer is custom and works very well, and mysql works very well and is not the bottleneck. The issue is is solr does not commit the documents fast enough. it hangs and then stops after 100 million rows.

We can easily get 100,000 rows per second from all parts, apart from the actual submit / index to solr. More information can be found here:

[url removed, login to view]:changehistory-tabpanel

We need urgent help and advice from an expert who has used solr to index these sort of speeds / data before.

Thanks

Skills: Apache Solr, Big Data, Database Administration, Java, MySQL

See more: solr large index, apache solr indexing data, indexing solr, java indexing, get expert advice, solr java, solr apache, jira atlassian, large data, jira, 100 million, small jira, solr urgent, information system mysql java, java mysql information system, java data mysql, information system java mysql, need java expert help, apache solr help, java expert system, deal plugin, 200gb mysql, expert system java, urgent help, domainmanuk

About the Employer:
( 41 reviews ) dalts, United Kingdom

Project ID: #4761984

8 freelancers are bidding on average $1434 for this job

programmedf

Hi, I hope you are fine. You migh have heap and tranaction log problems. For explaination of my appraoch to slow solrCloud runs, kindly see Personal Message. Have a nice day. Kind Regards, ProgrammedFuture

$666 USD in 10 days
(24 Reviews)
4.8
hbxfnzwpf

I am very proficient in java. I have 12 years java developing experience. I have worked for 5 years. Please let expert help you.

$1111 USD in 15 days
(9 Reviews)
4.0
Bramfeld

Are you using autocommit for some particular reason? In your case it would make more sense to take ownership of this important step, and tune it according to your needs (Solr offers many ways to do that).

$650 USD in 10 days
(2 Reviews)
2.7
angan

I CAN DO IT

$5555 USD in 60 days
(9 Reviews)
2.5
pardeepkaursonia

Hello. Please check PMB

$500 USD in 10 days
(2 Reviews)
0.0
i7informatics

Hi, i can able to analyze your database and i will give better solution to your problem

$777 USD in 3 days
(0 Reviews)
0.0
on2it

Kia ora! [url removed, login to view] are a NZ based web and software design company who have seen you here on the freelance market and are really excited about working with you and treating you to the full service, professional ex More

$1546 USD in 30 days
(0 Reviews)
0.0
niketkumar

I am an experienced programmer with skills relevant to the project. I have worked for companies like Thoughtworks, Verisign, Yahoo and Netapp. Please have a look at my linkedin profile for the details. [url removed, login to view] More

$666 USD in 10 days
(0 Reviews)
0.0