Android string concat dando outofmemoryerror

I'm working on a project where I'm downloading packets of data over internet and parsing them in my application. Every packet has it's own structure and in the beginning I'm receiving only text data. But in some point I start to get binary packets with images and this kind of stuff and sometimes on some devices when the binary file is too big I'm getting this error. Actually it's never happend while I'm testing, but I'm getting some reports from users. Here is actually how I'm doing it :

reading response :

InputStream response = new BufferedInputStream(connection.getInputStream());

        int bytesRead = -1;
        byte[] buffer = new byte[30 * 1024];
        while ((bytesRead = response.read(buffer)) > 0 && stopThread) {
            byte[] buffer2 = new byte[bytesRead];
            System.arraycopy(buffer, 0, buffer2, 0, bytesRead);
            handleDataFromSync(buffer2);
        }

and parsing the data like this :

public void handleDataFromSync(byte[] buffer) {
    RPCPacket packet;
    String responseBody;

    while(!stopThread) return;
    try {
        responseBody = new String(buffer, "UTF-8");
        StringBuilder tmp = new StringBuilder(responseBody);
        totalBytesReceived += responseBody.length();

        if (tmpBuffer != null) {
            tmpBuffer = tmpBuffer.append(tmp);
            tmp = tmpBuffer;
        }
        int bufferLen = tmp.length();
        int lastLoc = 0;
        boolean gotPacket;
        boolean gotField;
        String thisPart = "";

        try {
            do {
                gotPacket = false;
                gotField = true;
                int needsSize = packetFieldSizes[tmpCurrentField - 1];
                if (tmpCurrentField == packetFieldSizes.length) {
                    needsSize = payloadSize;
                }
                if (needsSize > bufferLen - lastLoc) {
                    gotField = false;
                    String proba = tmp.substring(lastLoc);
                    tmpBuffer =  new StringBuilder(proba);
                    break;
                }
                thisPart = tmp.substring(lastLoc, lastLoc + needsSize);
                lastLoc += needsSize;
                if (gotField) {

                    switch (tmpCurrentField) {

                    case 1: {
                        long intVal = Long.parseLong(thisPart);
                        objectIdentificator = (int) intVal;
                        break;
                    }
                    case 2: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        pType = (short) intVal;
                        break;
                    }
                    case 3: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        operationType = (short) intVal;
                        break;
                    }
                    case 4: {
                        objectOId = thisPart;
                        break;
                    }
                    case 5: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        id = (int) intVal;
                        break;
                    }
                    case 6: {
                        long intVal = Long.parseLong(thisPart);
                        payloadSize = (int) intVal;
                        dataSize = (int) intVal;
                        break;
                    }
                    case 7: {
                        hashH = thisPart;
                        break;
                    }
                    case 8: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        dataType = (short) intVal;
                        break;
                    }
                    case 9: {
                        if (payloadSize != 0) {
                            byte[] tmpData = Base64.decode(thisPart);
                            first = tmpData;
                        }
                        break;
                    }
                    }

                    if (tmpCurrentField >= packetFieldSizes.length)
                        gotPacket = true;

                    if (gotPacket) {
                        Log.d("", "Gotpacket!");
                        packet = new RPCPacket(objectIdentificator,
                                RPCPacketType.getPacketTypeByValue(pType),
                                RPCOperationType.getByValue(operationType),
                                objectOId, id, dataSize, hashH,
                                RPCPacketDataType.getByValue(dataType),
                                first);
                        parseRPCPacket(packet);

                        myProgress++;
                        update();
                        Log.e("","myProgress : "+myProgress);
                        Log.e("","TOTAL PACKETS : "+RPCCommunicator.totalPackets);

                        // release temp fields
                        objectIdentificator = 0;
                        pType = 0;
                        operationType = 0;

                        objectOId = null;

                        id = 0;
                        dataSize = 0;

                        hashH = null;

                        dataType = 0;

                        first = null;

                        tmpCurrentField = 1;
                        payloadSize = 0;

                    } else {
                        tmpCurrentField++;
                    }
                }

                // you baad bad buffer
                assert (lastLoc <= bufferLen);

                if (isBad)
                    break;

            } while (true);

        } catch (IOException e) {
            e.printStackTrace();
            RPCCommunicator.writeLogs(e.toString(), "Synchronization" ,"handleDataFromSync");

        } finally {
            thisPart = null;
            tmp = null;
        }
    } catch (Exception e) {
        e.printStackTrace();
        RPCCommunicator.writeLogs(e.toString(), "Synchronization","handleDataFromSync");
    }

}

so the error is thrown in that line : tmpBuffer = tmpBuffer.concat(tmp); only when the image is too big and it's concatenating the string a few times to get the whole packet. I'm reading response in 30KB pieces, but I can receive images with 300, 400KB and etc.

So any kind of idea how can I get rid of this issue. I'm not really sure what can I use instead of this.

Gracias de antemano!

preguntado el 31 de enero de 12 a las 08:01

Empiece por usar un StringBuilder (o StringBuffer) instead of using Strings. Are you concat:ing the binary stuff into your tmp variable also? -

no, I'm concatenating only strings, but I'm reading the response in byte array, which I'm using to create resposeBody -

Well, if you give us a bit more info on the protocol in question (I assume it's pretty custom?) maybe we can see if there's a bit more efficient way to do it. -

Ok, Ican show you how I'm parsing the packets. Just edited the question.. Actually I've just change it with StringBuilder...but can't try it because it never happened while I'm using the app. -

2 Respuestas

Just like Jens said, just use a StringBuilder (http://developer.android.com/reference/java/lang/StringBuilder.html) or a StringBuffer (http://developer.android.com/reference/java/lang/StringBuffer.html) if you need synchronized calls from different threads.

In your current code every time you concat something to the String a new object is created, thus the memory problems can occur. When using a StringBuilder only one Object is used.

Respondido el 31 de enero de 12 a las 12:01

So, for a starter, the way you read your stream allocates too much memory, specifically this part:

byte[] buffer2 = new byte[bytesRead];
System.arraycopy(buffer, 0, buffer2, 0, bytesRead);
handleDataFromSync(buffer2);

Consider revising the API of handleDataFromSync(byte[] buffer) a handleDataFromSync(byte[] buffer, int start, int count) and do like this when you read you stream:

while ((bytesRead = response.read(buffer)) > 0 && stopThread) {
    handleDataFromSync(buffer, 0, bytesRead);
}

You can create a string from that using new String(buffer, offset, count, "UTF-8") in handleDataFromSync (just dumping bytes into a String in this way is not guaranteed to result in a properly decoded String if you including cualquier character that will use more than one octet in UTF-8, such as ÅÄÖ or other junk).

The string handling in your parsing is a bit too vague to judge - have you measured / printed how much tmpBuffer grows when parsing for instance?

In your case I'd consider using an InputStreamReader, and trying to revise the parsing altogether - is there any reason you must read 30kb chunks?

(Para su información, assert(..) is pretty much disabled in Android, check & throw an exception if you want to guard against stuff).

Respondido el 31 de enero de 12 a las 14:01

Estoy haciendo esto System.arraycopy(buffer, 0, buffer2, 0, bytesRead); , because when I receive the last packet from the stream, it can be less than 30kb, and in that situation my buffer will be filled to it's size with some non characters...as I saw that before starting usin' this. - Android-Droid

and forget to mention, I'm reading the stream in 30kb chunks, because I've never know how much data I'll download. In some situations it can be like 5-10MB, in other it can be like 300MB too. - Android-Droid

Well, you're not processing the received bytes - you're converting them into a String - and a string can be created from the "larger" 30 kb buffer if you supply an offset & count. The savings are not that great - less fragmentation and GC on the heap probably. - Jens

I'll read your answer again carefully. Because actually in some cases when I'm downloading a big stream like 50MB it ca take line 2-3minutes to downloa it and to parse it..which I'm trying to avoid. - Android-Droid

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.