Descarga de partes de archivos usando HttpWebRequest C#

I am trying to download a 100GB file using HttpWebRequest. The download will be split into parts depending on a preset part size. Below is the code I use to download the file:

private static void AddRangeHeaderHack(WebHeaderCollection headers, long start, long end)
        {
            // Original workaround by Eric Cadwell, code taken from
            // https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93714
            Type type = headers.GetType();

            System.Reflection.MethodInfo setAddVerified = type.GetMethod("SetAddVerified",
                System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.FlattenHierarchy
                );

            string rangeHeaderValue = String.Format("bytes={0}-{1}", start, end);
            if (setAddVerified != null)
                setAddVerified.Invoke(headers, new object[] { "Range", rangeHeaderValue });
        }

        private ulong GetRemoteFileSize(string URI)
        {
            ulong size = 0;
            HttpWebRequest req = null;
            try
            {
                req = (HttpWebRequest)WebRequest.Create(URI);
                using (var res = (HttpWebResponse)req.GetResponse())
                {
                    size = (ulong)res.ContentLength;
                    res.Close();
                }
            }
            catch (Exception ex)
            {

            }

            if (req != null)
            {
                try
                {
                    req.Abort();
                    req = null;
                }
                catch (Exception)
                {

                }
            }

            return size;
        }

        private int DownloadFromLink(string sSource, string sDestination)
        {
            int nRetryCount = 0;
            int nMaxRetry = 5;
            var lastProgress = DateTime.Now;
            ulong offset = 0;
            var bRetrying = false;
            var bResumable = false;

            var fileSize = GetRemoteFileSize(sSource);
            if (fileSize > 0)
                bResumable = true;

            while (true)
            {
                HttpWebRequest webRequest = null;
                try
                {
                    try
                    {
                        bRetrying = false;
                        do
                        {
                            try
                            {
                                if (bDownloadAbort)
                                {
                                    return -1;
                                }

                                webRequest = (HttpWebRequest)WebRequest.Create(sSource);
                                webRequest.Timeout = 3600000;

                                if (offset > 0)
                                {
                                    AddRangeHeaderHack(webRequest.Headers, (long)offset, (long)fileSize);
                                }

                                // Retrieve the response from the server
                                using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
                                {

                                    var acceptRanges = String.Compare(webResponse.Headers["Accept-Ranges"], "bytes", true) == 0;

                                    // Open the URL for download 
                                    using (var streamResponse = webResponse.GetResponseStream())
                                    {
                                        if (streamResponse != null)
                                        {
                                            // Create a new file stream where we will be saving the data (local drive)
                                            using (var streamLocal = new FileStream(sDestination, offset>0?FileMode.Append:FileMode.Create, FileAccess.Write, FileShare.ReadWrite))
                                            {
                                                // It will store the current number of bytes we retrieved from the server
                                                int bytesSize = 0;
                                                // A buffer for storing and writing the data retrieved from the server
                                                byte[] downBuffer = new byte[/*16384*/ 1024 * 1024];
                                                bool binitialtry = true;
                                                int nRetries = 0;

                                                if (offset > 0)
                                                {
                                                    streamLocal.Seek((long)offset, SeekOrigin.Begin);
                                                }

                                                // Loop through the buffer until the buffer is empty
                                                while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0 
                                                    || (File.Exists(sDestination) && (offset < (ulong)fileSize) && nRetries < 5 && bResumable))
                                                {
                                                    if (binitialtry && bytesSize == 0)
                                                    {
                                                        binitialtry = false;
                                                    }

                                                    if (!binitialtry && bytesSize == 0)
                                                    {
                                                        nRetries++;
                                                        bRetrying = nRetries<5;
                                                        break;
                                                    }

                                                    if (bDownloadAbort)
                                                    {
                                                        try { streamLocal.Close(); }
                                                        catch { }

                                                        return;
                                                    }

                                                    try
                                                    {
                                                        // Write the data from the buffer to the local hard drive
                                                        streamLocal.Write(downBuffer, 0, bytesSize);
                                                        offset += (ulong)bytesSize;
                                                    }
                                                    catch (IOException ex)
                                                    {
                                                        if (streamResponse != null)
                                                            streamResponse.Close();

                                                        if (streamLocal != null)
                                                            streamLocal.Close();

                                                        if (webRequest != null)
                                                            webRequest.Abort();

                                                        return -1;
                                                    }

                                                    Interlocked.Add(ref actualDownloaded, bytesSize);
                                                }

                                                // When the above code has ended, close the streams
                                                if (streamResponse != null)
                                                    streamResponse.Close();

                                                if (streamLocal != null)
                                                    try { streamLocal.Close(); }
                                                    catch { }

                                                if (webRequest != null)
                                                    webRequest.Abort();

                                                if (webRequest != null)
                                                    wcDownload.Dispose();

                                                streamLocal.Close();
                                            }
                                            streamResponse.Close();
                                        }
                                    }
                                    webResponse.Close();
                                }

                                if(!bRetrying)
                                    break;
                            }
                            catch (IOException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                if (wcDownload != null)
                                    wcDownload.Dispose();

                                if (nRetryCount <= nMaxRetry)
                                {
                                    Thread.Sleep(10000);
                                    nRetryCount++;
                                    bRetrying = true;
                                }
                                else
                                {
                                    break;
                                }
                            }
                            catch (UnauthorizedAccessException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                break;
                            }
                            catch (WebException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                if (wcDownload != null)
                                    wcDownload.Dispose();

                                if (nRetryCount <= nMaxRetry)
                                {
                                    Thread.Sleep(10000);
                                    nRetryCount++;
                                    bRetrying = true;
                                }
                                else
                                {
                                    break;
                                }
                            }
                            finally
                            {


                            }

                        } while (bRetrying);


                    }
                    catch (Exception ex)
                    {
                        break;
                    }
                }
                catch
                {
                    break;
                }

                if(!bRetrying)
                    break;
            }
        }

If I try to download the file in 1 part, with out adding the range header, the code runs smoothly and the file downloads normally. When I add a range header, say from 10GB to 15GB or frankly any value, the code reaches streamResponse.Read and hangs there for several minutes then it throws an exception:

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

When the code retries the connection after the exception, the download resumes normally and the client is able to read data from the stream.

Can someone help me determine why such thing is happening?

Just to clear the matter about the server, the file is currently hosted on an Amazon S3 server, and the download is done from a generated direct link.

preguntado el 28 de mayo de 14 a las 13:05

2 Respuestas

It could be a server setting, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4

Los clientes que utilizan conexiones persistentes DEBEN limitar el número de conexiones simultáneas que mantienen a un servidor determinado. Un cliente de un solo usuario NO DEBE mantener más de 2 conexiones con ningún servidor o proxy. Un proxy DEBE usar hasta 2 * N conexiones a otro servidor o proxy, donde N es el número de usuarios activos simultáneamente. Estas pautas están destinadas a mejorar los tiempos de respuesta HTTP y evitar la congestión.

Try FDM, and see if it has a problem. http://www.freedownloadmanager.org/

contestado el 28 de mayo de 14 a las 14:05

I thought about that but the problem happens even if a single connection is established to the server. No matter what I try I cannot get it to download from a specific range the first time. It always take two tries to start reading from the stream. And I have tried downloading the file using a download manager, I used IDM, and it managed to download the file with no problems while maintaining 8 connections with the server. - Zaid Amir

Now I wouldn't mind it taking a couple of retries to connect though its the hang that is driving me nuts. The first attempt to read from a range stream hangs for almost 30 minutes before throwing the exception mentioned the question. - Zaid Amir

I don’t know how to download a file in parts using HttpWebRequest, but I found this ejemplo online to build an own implementation. The article is about the HttpClient in C#. There is also complete code and project you can find in the download section of this page.

The Problem is that not all server support partial download. So NuGet packages can be used that handle the exceptions e.g.: https://www.nuget.org/packages/downloader or https://www.nuget.org/packages/Shard.DonwloadLibrary. These Libraries will handle the chunks and convert them back into a readable file or stream.

Descargador:

var downloader = new DownloadService(new()
    {
        ChunkCount = 8,
    });
    string file = @"Your_Path\fileName.zip";
    string url = @"https://file-examples.com/fileName.zip";
    await downloader.DownloadFileTaskAsync(url, file);

or Download Library:

    string file = "Your_Path";
    string url = "https://file-examples.com/fileName.zip";
    var downloader = new LoadRequest(url,new()
    {
        Chunks = 8,
        DestinationPath= file,
    });

    await downloader.Task;

¡Espero poder ayudar!

Respondido el 16 de diciembre de 22 a las 11:12

No es la respuesta que estás buscando? Examinar otras preguntas etiquetadas or haz tu propia pregunta.