gem mirror should also download the /versions file and the /info/ directory

[iBug]

According to issues like:

• Rubygems 速度缓慢 ustclug/discussions#438 (Chinese)
• Consider reverse-proxying api.rubygems.org/api/v1/dependencies tuna/issues#374

A full RubyGems mirror generated by gem mirror is not easy to consume - Both gem install and bundle fetches tons of gemspec.rz files if a faster index is not found (previously /api/v1/dependencies, now superseded by /versions + /info/), making a local mirror actually much slower even under terrible availability of rubygems.org (e.g. from mainland China).

If gem mirror could also fetch the versions file and populate the /info/ directory, downstream mirror sites will be much easier to use.

[segiddins]

That would be a great improvement, I'd be happy to review a PR if you're able to pull one together!

[iBug]

I took a look at the existing code and it seems too entangled for me to handle. I hope someone from Ruby could take on this.

My ideas are:

• Fetch /versions unconditionally and either:
  • Generate /versions.md5sum file so we (mirror sites) can serve its content as the ETag value for the /versions file, to maintain compatibility with Bundler 2.4.
  • Save the ETag server header to /versions.md5sum which may be even more compatible.
• For every gem, fetch /info/#{gem.name} if either:
  • The target file is missing.
  • A new version of the gem is being fetched.
• For every info file, delete it if deletion is enabled and the corresponding gem has gone.

[89ao]

@iBug Share one of our practices, which is to redirect the /info/ directory to the official address, so that no matter what package is downloaded and installed, the original metadata will be accessed.

[iBug]

According to tuna/issues#2048, a mirror created by gem mirror without additional care is unusable with gem.