Thrift / Proto ¿Cómo?

I need to serialize a large array of object of 100000 to a file, but during de-serialization I would like to read chuck of 1000 objects per read.

Currently I don't find any good examples of either Thrift or Protocol Buffer for C#. Please can someone help me out.


preguntado el 10 de marzo de 12 a las 17:03

2 Respuestas

Well protobuf-csharp-port has MessageStreamIterator y MessageStreamWriter which allow you to write out a sequence and then iterate over them as you deserialize. Then you can use LINQ to Objects to get 1000 or whatever you want.

respondido 10 mar '12, 17:03

If you use protobuf-net, there is a method DeserializeItems (or similar) that provides an IEnumerable<T> implementation, non-buffered, allowing you to consume items as you like (suitable for LINQ "Take" etc). This works perfectly if the original serialized object is a list/array of objects. It is also fully compatible with the DeserializeWithLengthPrefix method, which reads a single object from a stream without over-reading (as log as it is length-prefixed, which is the case if an array/list of items is serialized), allowing you to do the same thing in, say, a "while" loop. Let me know if you want more complete examples.


// write
YourType[] array = ...;
Serializer.Serialize(destination, array);

// read
List<YourType> batch = new List<YourType>(1000);
foreach(var item in Serializer.DeserializeItems<YourType>(source)) {
    if(batch.Count == 1000) {
if(batch.Count != 0) ProcessBatch(batch);

Note also that protocol buffers is appendable, so to add more data to a file you just seek to the end of the file and start writing more data.

respondido 10 mar '12, 17:03

Are our two solutions compatible with each other, out of interest? - jon skeet

Thanks for your quick response, it will be great if you can provide complete example. - tabreaz

@Jon mine is, per item: an opcional tag as varint, typically "1", then the length as varint, then the payload. Is that compatible? - marc gravell

@user1261267 no problem - I'm on mobile currently; can add full/tested example later, but rough estimate added - marc gravell

@Jon oops, I meant "tag decorated as a string wire-type", and note that it also supports big/little-endian 4-byte int length prefix (without tag/wire-type), as lots of people seem to use that. - marc gravell

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.