La importación de datos falla para datos grandes en SOLR

I have setup SOLR. I used to import data (1 lakh) into solr index which works properly.

Now i am trying to import large data (10+ lakhs records) into solr index. i have setup data-config.xml then i start full-import but it fails and give error mentioned below

La indexación falló. Revirtió todos los cambios.

I store solr start log in solr.err file which gives error:

Jul 31, 2012 10:19:33 AM org.apache.solr.update.DirectUpdateHandler2 rollback
SEVERE: Ignoring Error when closing connection
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4749)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2627)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:841)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:681)
    ... 6 more
Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback
SEVERE: Ignoring Error when closing connection
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2627)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:841)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:681)
    ... 6 more
Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback

SEVERE: Ignoring Error when closing connection at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)

Jul 31, 2012 10:19:44 AM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
Jul 31, 2012 10:19:44 AM org.apache.solr.handler.dataimport.JdbcDataSource closeConnection
SEVERE: Ignoring Error when closing connection

com.mysql.jdbc.PacketTooBigException: Packet for query is too large (7959657 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:637)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:420)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3118)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2288)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2709)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2677)
    at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4863)
    at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4749)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4358)
    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1607)

@ ataConfig.java:342) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:383) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:445) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:426)

I use 4 core processor with 4GB RAM.

What mysql/java configuration changes required to resolve this issue?

preguntado el 31 de julio de 12 a las 11:07

Didn't you see that MySQL reported "You can change this value on the server by setting the max_allowed_packet' variable"? That's all you need to do. -

1 Respuestas

I guess this has more to do with Mysql than with Solr, as mysql jdbc driver is having the issue. Try this workaround here, that just uppers the limit of data mysql can handle.

Respondido 31 Jul 12, 12:07

Even after setting max_allowed_packet it was giving java heap space issue which is resolved by increasing heap size while starting solr with java -Xmx1582m -jar start.jar - John Smith

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.