How Cloudflare Broke My Build and How I Fixed It
Mysterious Build Failure
About a month ago, my open-source project EntityFramework.Exceptions failed to build on AppVeyor, the continuous integration service that I have been using for several years. I hadn’t made any changes that could have caused the build to fail, so I set out to investigate the cause of the failure.
It turned out that the build failed because it failed to upload source code coverage data to coveralls.io. The error message was very minimal:
BadRequest - <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
I checked
coveralls status page, but there wasn’t any ongoing incident, so I reached out to coveralls support. Coveralls support responded that they had recently migrated their services to different servers and were now using Cloudflare as a service provider but these changes should have been transparent to the end users. To ensure this wasn’t a DNS issue, I added ping -n 1 coveralls.io to the appVeyor.yml file. The resolved IP address was correct, so the issue wasn’t related to DNS.
Coveralls support tried to check the cause of the Bad Request error in their logs but surprisingly my attempts to upload the coverage data didn’t show up. This meant that the Bad Request didn’t originate from coveralls.io and was coming from Cloudflare. Why was Cloudflare suddenly rejecting my attempts to upload coverage that was working flawlessly for the last couple of years?
Finding the Root Cause of the Issue
Now that I knew that the failure was Cloudflare’s fault, I asked the coveralls.io team to get in touch with Cloudflare support to find out the cause of the error response. I downloaded
coveralls.net (Coveralls uploader for .Net Code coverage) to my local machine, attempted to submit the coverage data, and captured the request with
Fiddler. I saved the HTTP request as a har file and sent it to coveralls.io support, hoping Cloudflare would help solve the issue. Unfortunately, Cloudflare was not helpful at all.
My next step, as suggested by James from coveralls.io, was to submit the coverage file with curl and see if that worked. I ran the following command: curl https://coveralls.io/api/v1/jobs --form "json_file=@uploadedCoverage.json" and to my surprise, the coverage file was successfully received by coveralls.io. Now that I was able to submit coverage data with curl, all I had to do, was to compare the HTTP requests issued by curl and coveralls.net to find the difference between them that was causing the failure.
The HTTP request looked like this (request bodies omitted):
coveralls.net HTTP request:
POST https://coveralls.io/api/v1/jobs HTTP/1.1
Host: coveralls.io
Content-Type: multipart/form-data; boundary="e5d8bb1a-4ab2-4bb8-9d8b-152c5b704cfa"
Content-Length: 3093
--e5d8bb1a-4ab2-4bb8-9d8b-152c5b704cfa
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=json_file; filename=coverage.json; filename*=utf-8''coverage.json
curl HTTP request:
POST https://coveralls.io/api/v1/jobs HTTP/1.1
Host: coveralls.io
User-Agent: curl/7.83.1
Accept: */*
Content-Length: 3080
Content-Type: multipart/form-data; boundary=------------------------c8eebbf0920b1cc0
--------------------------c8eebbf0920b1cc0
Content-Disposition: form-data; name="json_file"; filename="uploadedCoverage.json"
Content-Type: application/octet-stream
There are several differences between these requests, but can you guess what was causing Cloudflare to fail? It turned out that Cloudflare returned Bad Request because of double quotes around the boundary directive value in the Content-Type header!
Content-Type boundary with double quotes is perfectly valid, so why does Cloudflare treat it as an error? Unfortunately, I don’t have an answer to this question. The
pull request to handle quotes was merged more than two years ago, so either Cloudflare uses an older version of
nginx upload module, or Cloudflare uses another module that cannot handle double quotes around the boundary directive.
Fixing Coveralls Uploader
Now that I knew what was causing the failure, it was easy to fix the coveralls.net:
var boundary = formData.Headers.ContentType?.Parameters.FirstOrDefault(o => o.Name == "boundary");
if (boundary != null)
{
boundary.Value = boundary.Value?.Replace("\"", string.Empty);
}
I submitted a pull request to the coveralls.net repo, and as soon as it was merged, my build was green again!
