Error when pulling ACR Helm repository (index and chart)

Question

We use ACR to store Docker images and Helm charts. We use Helm chart with the Helm repository (not the OCI registry).
The URI for the helm repository is : https://contoso.azurecr.io/helm/v1/repo

We have an operator (Flux v2) running inside an kubernetes cluster (AKS) that regularly pulls some charts, and the list of available charts. But sometimes it has error responses from ACR, then some success responses, and later again some error responses and so on without doing anything.

Here is an error from logs from Flux v2 component when pulling a Helm chart :
{"level":"error","ts":"2020-11-10T11:45:33.258Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"HelmChart","controller":"helmchart","name":"default-cdn","namespace":"cd","error":"failed to fetch https://contoso.azurecr.io/helm/v1/repo/_blobs/cdn-1.0.0.tgz : 500 Internal Server Error","errorVerbose":"failed to fetch https://contoso.azurecr.io/helm/v1/repo/_blobs/cdn-1.0.0.tgz : 500 Internal Server Error\nhelm.sh/helm/v3/pkg/getter.(*HTTPGetter).get

And another error when getting the charts list :
{"level":"error","ts":"2020-11-10T11:46:39.273Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"source.toolkit.fluxcd.io","reconcilerKind":"HelmRepository","controller":"helmrepository","name":"contoso","namespace":"cd","error":"failed to download repository index: failed to fetch https://contoso.azurecr.io/helm/v1/repo/index.yaml : 500 Internal Server Error"}

What is the cause of this error and how can I get more details about it ?

Answer 1

We checked for your ACR and found that all 500 errors are linked with invalid header field value "xxxxxxxxxxxxxxxxxx\n" for key Origin-User
The mitigation is to trim the ending white space of the username before any auth operations.

We saw this issue in instances when user uses scripts to do login and auth related operations. Due to the line ending differences such as \n and \r\n, the character \r was send as input to the username field.

Let me know if you if you have further questions.

---

Please Accept as answer if it helped, so that it can help others facing the same issue.

Answer 2

@fredgate

Not really sure I understood what you solution was, where did you fix the problem?

I've run into the same issue using Flux v2 and a HelmRepository backed by ACR.

Answer 3

I should add that my problem was related to the base64 encoding of username/password.

I did like this:

echo "username" | base64

What I didn't know is that echo adds a newline at the end, and the encoded strings ends with: Cg==.

What I had to do was this instead

echo -n "username" | base64