Transfer closed with outstanding read data remaining ошибка

Usually, the cURL error ‘cURL 18 transfer closed with outstanding read data remaining’ occurs while retrieving data from a URL using a cURL.

Here at Bobcares, we have seen several such cURL related errors as part of our Server Management Services for web hosts and online service providers.

Today we’ll take a look at the causes for this error and see the fix.

More about ‘cURL 18 transfer closed with outstanding read data remaining’ error

Sometimes, the file we transfer will be smaller or larger than expected. Such cases arise when the server initially reports an expected transfer size, and then delivers data that doesn’t match the previously sent size.

In short, this error is related to content-length.

cURL error 18 can be described as below:

CURLE_PARTIAL_FILE

It means a Partial file i.e. only a part of the file was transferred.

Different causes and fixes for ‘cURL 18 transfer closed with outstanding read data remaining’ error

Now let’s see the different causes for this error. Also, we shall see how our Support Engineers fix it.

1. An incorrect Content-Length header was sent by the peer.

If an incorrect Content-Length header is been sent then the best option is to allow the cURL to set the length by itself. This will avoid the issues that might arise after setting the wrong size.

Moreover, we can fix this by suppressing the ‘Expect: 100-continue’ header that cURL usually sends.

curl_setopt($curl, CURLOPT_HTTPHEADER, array(‘Expect:’));

2. The connection gets timed-out as keep-alives were not sent to keep the connection going on.

To fix this issue, add the –keepalive-time.

For instance,

–keepalive-time 2

This option will set a time a connection will need to remain idle before sending the keepalive probes and the time between individual keepalive probes. However, if we use –no-keepalive then this option has no effect.

In case, if we use this option several times then the last one will be used. If you don’t specify the value then it defaults to 60 seconds.

In PHP cURL, the –keepalive-time option is available from the PHP version 5.5. You can use it as follows:

curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1);
curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2);

[Need any further assistance in fixing cURL errors? – We are here to help you.]

Conclusion

In short, this error occurs while retrieving data from a URL using a cURL. Today, we saw how our Support Engineers fix this error.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;

when retrieving data from a URL using curl, I sometimes (in 80% of the cases) get

error 18: transfer closed with outstanding read data remaining

Part of the returned data is then missing. The weird thing is that this does never occur when the CURLOPT_RETURNTRANSFER is set to false, that is the curl_exec function doesn’t return the data but displays the content directly.

What could be the problem? Can I set some of the options to avoid such behaviour?

The error string is quite simply exactly what libcurl sees: since it is receiving a chunked encoding stream it knows when there is data left in a chunk to receive. When the connection is closed, libcurl knows that the last received chunk was incomplete. Then you get this error code.

There’s nothing you can do to avoid this error with the request unmodified, but you can try to work around it by issuing a HTTP 1.0 request instead (since chunked encoding won’t happen then) but the fact is that this is most likely a flaw in the server or in your network/setup somehow.

Seeing this error during the use of Guzzle as well. The following header fixed it for me:

'headers' => [
    'accept-encoding' => 'gzip, deflate',
],

I issued the request with Postman which gave me a complete response and no error.
Then I started adding the headers that Postman sends to the Guzzle request and this was the one that fixed it.

I had the same problem, but managed to fix it by suppressing the ‘Expect: 100-continue’ header that cURL usually sends (the following is PHP code, but should work similarly with other cURL APIs):

curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));

By the way, I am sending calls to the HTTP server that is included in the JDK 6 REST stuff, which has all kinds of problems. In this case, it first sends a 100 response, and then with some requests doesn’t send the subsequent 200 response correctly.

Encountered similar issue, my server is behind nginx.
There’s no error in web server’s (Python flask) log, but some error messsage in nginx log.

[crit] 31054#31054: *269464 open() «/var/cache/nginx/proxy_temp/3/45/0000000453» failed (13: Permission denied) while reading upstream

I fixed this issue by correcting the permission of directory:

/var/cache/nginx

I had this problem working with pycurl and I solved it using

c.setopt(pycurl.HTTP_VERSION, pycurl.CURL_HTTP_VERSION_1_0) 

like Eric Caron says.

I’ve solved this error by this way.

$ch = curl_init ();
curl_setopt ( $ch, CURLOPT_URL, 'http://www.someurl/' );
curl_setopt ( $ch, CURLOPT_TIMEOUT, 30);
ob_start();
$response = curl_exec ( $ch );
$data = ob_get_clean();
if(curl_getinfo($ch, CURLINFO_HTTP_CODE) == 200 ) success;

Error still occurs, but I can handle response data in variable.

I got this error when i was accidentally downloading a file onto itself.
(I had created a symlink in an sshfs mount of the remote directory to make it available for download, forgot to switch the working directory, and used -OJ).

I guess it won’t really »help« you when you read this, since it means your file got trashed.

I got this error when running through a nginx proxy and I was running nginx under the user-id daemon instead of the user id nginx.

This means some of nginx’s scratch directories weren’t accessible / writable.

Switching from user daemon; to user nginx; fixed it for me.

I had this same problem. I tried all of these solutions but none worked. In my case, the request was working fine in Postman but when I do it with curl in php I get the error mentioned above.

What I did was check the PHP code generated by Postman and replicate the same thing.

First the request is set to use Http version 1.1
And the second most important part is the encoding for me.

Here is the code that helped me

curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

If I remove the CurlOpt Encoding I get back the error.

remote: Counting objects: 66352, done.
remote: Compressing objects: 100% (10417/10417), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

Ответы:

Cloning into 'large-repository'...
remote: Counting objects: 20248, done.
remote: Compressing objects: 100% (10204/10204), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining 
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

$ git clone http://github.com/large-repository --depth 1
$ cd large-repository
$ git fetch --unshallow

git clone username@mydomain.com:my_group/my_repository.git

git config --global http.postBuffer 524288000

remote: Counting objects: 182, done.
remote: Compressing objects: 100% (149/149), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

git clone https://username@bitbucket.org/repositoryName.git --depth 1

git config --global core.compression 0

git clone --depth=1 <url>

cd <shallow cloned project dir>

git fetch --depth=N, with increasing N

git fetch --depth=4

git fetch --depth=100

git fetch --depth=500

git fetch --unshallow 

git clone https://github.com/vaibhavjain2/xxx.git - Avoid
git clone git@github.com:vaibhavjain2/xxx.git - Correct

#Linux:
export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

#Windows
set GIT_TRACE_PACKET=1
set GIT_TRACE=1
set GIT_CURL_VERBOSE=1

git fetch --all  or git clone 

    

git fetch origin BranchName 

cd [dir]
git init
git clone [your Repository Url]