[temporary] chunked upload disabled (could not push size > 1024)

[yuvalturg]

Hi,

Whenever I try to push something larger than 1023, it fails as follows:

root@yuval:~# python3 -c 'open("foo", "wb").write(bytes(1024*"a", "ascii"))'
root@yuval:~# oras-py push -i --disable-path-validation registry.container-registry.svc.cluster.local:5000/test/foo:123 ~/foo 
blob upload invalid
Issue with http://registry.container-registry.svc.cluster.local:5000/v2/test/foo/blobs/uploads/fcda98d7-1086-48f2-bd8a-a3f41212f2b1?_state=bMUNu4I6M65BujbiZ80Qd1CnpzS8eEdNXwQkX1Pko0x7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJmY2RhOThkNy0xMDg2LTQ4ZjItYmQ4YS1hM2Y0MTIxMmYyYjEiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjQ6NDkuNDEwNTYzNTY5WiJ9&digest=sha256%3A2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a:
Not Found
root@yuval:~# 

This is the log on the registry's side:

time="2022-09-22T16:25:13.641318415Z" level=info msg="response completed" go.version=go1.11.2 http.request.contenttype="application/octet-stream" http.request.host="registry.container-registry.svc.cluster.local:5000" http.request.id=29d94599-bd34-473e-97f1-299a9eeea5e6 http.request.method=POST http.request.remoteaddr="10.1.96.136:54946" http.request.uri="/v2/test/foo/blobs/uploads/" http.request.useragent="python-requests/2.28.1" http.response.duration=12.64265ms http.response.status=202 http.response.written=0 
10.1.96.136 - - [22/Sep/2022:16:25:13 +0000] "POST /v2/test/foo/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.28.1"
time="2022-09-22T16:25:13.658398142Z" level=info msg="response completed" go.version=go1.11.2 http.request.contenttype="application/octet-stream" http.request.host="registry.container-registry.svc.cluster.local:5000" http.request.id=b193e768-3131-4d4c-8c2f-dd52023512e4 http.request.method=PATCH http.request.remoteaddr="10.1.96.136:54946" http.request.uri="/v2/test/foo/blobs/uploads/ab44b745-df06-485e-a6e0-1457ebfe117b?_state=6bmyqPyB_aox0XrXFHE-SvHnbvRZyO2fTU4bZOklkQR7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJhYjQ0Yjc0NS1kZjA2LTQ4NWUtYTZlMC0xNDU3ZWJmZTExN2IiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjU6MTMuNjMxNDM0NzEyWiJ9" http.request.useragent="python-requests/2.28.1" http.response.duration=13.321217ms http.response.status=202 http.response.written=0 
10.1.96.136 - - [22/Sep/2022:16:25:13 +0000] "PATCH /v2/test/foo/blobs/uploads/ab44b745-df06-485e-a6e0-1457ebfe117b?_state=6bmyqPyB_aox0XrXFHE-SvHnbvRZyO2fTU4bZOklkQR7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJhYjQ0Yjc0NS1kZjA2LTQ4NWUtYTZlMC0xNDU3ZWJmZTExN2IiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjU6MTMuNjMxNDM0NzEyWiJ9 HTTP/1.1" 202 0 "" "python-requests/2.28.1"
time="2022-09-22T16:25:13.662331644Z" level=error msg="upload resumed at wrong offest: 1024 != 0" go.version=go1.11.2 http.request.host="registry.container-registry.svc.cluster.local:5000" http.request.id=fca33ad6-2372-4ba1-b953-d0d7b70871eb http.request.method=PUT http.request.remoteaddr="10.1.96.136:54946" http.request.uri="/v2/test/foo/blobs/uploads/ab44b745-df06-485e-a6e0-1457ebfe117b?_state=6bmyqPyB_aox0XrXFHE-SvHnbvRZyO2fTU4bZOklkQR7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJhYjQ0Yjc0NS1kZjA2LTQ4NWUtYTZlMC0xNDU3ZWJmZTExN2IiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjU6MTMuNjMxNDM0NzEyWiJ9&digest=sha256%3A2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a" http.request.useragent="python-requests/2.28.1" vars.name="test/foo" vars.uuid=ab44b745-df06-485e-a6e0-1457ebfe117b 
10.1.96.136 - - [22/Sep/2022:16:25:13 +0000] "PUT /v2/test/foo/blobs/uploads/ab44b745-df06-485e-a6e0-1457ebfe117b?_state=6bmyqPyB_aox0XrXFHE-SvHnbvRZyO2fTU4bZOklkQR7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJhYjQ0Yjc0NS1kZjA2LTQ4NWUtYTZlMC0xNDU3ZWJmZTExN2IiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjU6MTMuNjMxNDM0NzEyWiJ9&digest=sha256%3A2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a HTTP/1.1" 404 76 "" "python-requests/2.28.1"
time="2022-09-22T16:25:13.666498169Z" level=error msg="response completed with error" err.code="blob upload invalid" err.message="blob upload invalid" go.version=go1.11.2 http.request.host="registry.container-registry.svc.cluster.local:5000" http.request.id=fca33ad6-2372-4ba1-b953-d0d7b70871eb http.request.method=PUT http.request.remoteaddr="10.1.96.136:54946" http.request.uri="/v2/test/foo/blobs/uploads/ab44b745-df06-485e-a6e0-1457ebfe117b?_state=6bmyqPyB_aox0XrXFHE-SvHnbvRZyO2fTU4bZOklkQR7Ik5hbWUiOiJ0ZXN0L2ZvbyIsIlVVSUQiOiJhYjQ0Yjc0NS1kZjA2LTQ4NWUtYTZlMC0xNDU3ZWJmZTExN2IiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDktMjJUMTY6MjU6MTMuNjMxNDM0NzEyWiJ9&digest=sha256%3A2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a" http.request.useragent="python-requests/2.28.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.788669ms http.response.status=404 http.response.written=76 vars.name="test/foo" vars.uuid=ab44b745-df06-485e-a6e0-1457ebfe117b 

Looks like a problem with _chunked_upload. When I modify the code to use _put_upload, it works.

Any ideas?

[vsoch]

I can look into this - but is there a particular reason to use oras-py for its command line client? I'd typically suggest using the Go client for this - oras-py is mostly intended for using from within Python.

[yuvalturg]

Thanks for the quick response - I am using it inside python actually. I was getting this weird error from my script so I just used oras-py for debugging. I did go over the go code real quick, and didn't see any reference to chunked uploads.

[vsoch]

okay reproduced!

[yuvalturg]

I looked a little further, the last PUT is missing the headers (Content-Length, Content-Range etc..)

[vsoch]

yeah I'm just seeing that - trying to load the distribution spec page to check details but my browser is timing out (I hope GitHub isn't going down!)

[vsoch]

yep here we go!

[vsoch]

It's actually not the last PUT that is the issue - it's the first chunk upload. Here are the headers used:

{'Content-Range': '0-1023',
 'Content-Length': '1024',
 'Content-Type': 'application/octet-stream'}

[vsoch]

The error I see is:

upload resumed at wrong offset: 1024 != 0

which is strange because we ask for the original URL with Content-Length 0, and then the range starts at 0. But for some reason this seems to think it's starting at 1024.

[vsoch]

It seems to be comparing the size to the offset, which doesn't make sense to me https://github.com/distribution/distribution/blob/78b9c98c5c31c30d74f9acb7d96f98552f2cf78f/registry/handlers/blobupload.go#L304-L313

[vsoch]

https://github.com/distribution/distribution/blob/78b9c98c5c31c30d74f9acb7d96f98552f2cf78f/registry/client/blob_writer.go#L73-L110

[wolfv]

Do you do chunked uploads? Because afaik they are not supported on Github

[vsoch]

That shouldn't matter I'm running a registry locally to reproduce the error.

[wolfv]

Or at least they were not supported at some point in the past :)

[vsoch]

I'm really not sure what to try here - this (I think) is following the spec to a T and the error doesn't make sense. I also thought it worked in the past, so I'm a bit dumbfounded. Maybe something changed?

[yuvalturg]

I honestly have no idea, I started using Oras just a few days ago, and all my prototypes were using small files :D

[vsoch]

If it's the case that some registries don't even support it, should we just disable it for now? And come back when there is a huge file that a registry cannot upload (needing chunked?)

[vsoch]

This is my proposal for now: #45

[yuvalturg]

Well, we're not using any registry, we're using the one that ships with microk8s which is an offical docker registry (docker.io/library/registry:2.7.1)

[yuvalturg]

This is my proposal for now: #45

That's what I did locally as well - just disabled the chunks_upload :)

[vsoch]

My suggestion is to to merge disabling chunked for the time being (which will get most reasonable sized uploads working) and come back to this when it presents an issue. It's not perfect, but I've taken a look and I can't figure out what the issue is - and most of the main maintainers on this project are focused on the Go bits and largely ignore work over here. So I'm going to go ahead and merge the patch, and we can come back in the future when someone has more insight to why the registry is not happy with the header shown above. Thanks @yuvalturg !

[yuvalturg]

That's great, thank you very much @vsoch !!

[vsoch]

Title is updated to reflect that this is now considered a bug! If anyone in the future sees this issue, we have the code to implement the chunked upload but ran into the issues above, and to re-enable it we will need to understand why the registry does not like our header specification.