Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add golang package mirroring #165

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions pre_commit_mirror_maker/golang/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module github.com/pre-commit/dummy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use the word "placeholder" instead as to avoid ableist language


go 1.19
Empty file.
3 changes: 3 additions & 0 deletions pre_commit_mirror_maker/golang/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package main

func main() {{}}
18 changes: 18 additions & 0 deletions pre_commit_mirror_maker/languages.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import subprocess
import urllib.request

import lxml.html
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should not need an external dependency to read xml -- there's an xml stdlib module

from packaging import version


Expand Down Expand Up @@ -31,6 +32,21 @@ def rust_get_package_versions(package_name: str) -> list[str]:
return list(reversed([version['num'] for version in resp['versions']]))


def golang_get_package_versions(package_name: str) -> list[str]:
url = f'https://pkg.go.dev/{package_name}?tab=versions'
resp = urllib.request.urlopen(url)
versions = lxml.html.parse(resp).xpath(
"//a[@class='js-versionLink']//text()",
)
return [str(version) for version in versions[::-1]]
Comment on lines +36 to +41
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should not be parsing html to get the answers here -- this needs to come from an actual documented api

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, go doesn't provide an actual documented API, the only workaround was to scrape it : /

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't they just git tags?



def golang_get_additional_dependencies(
package_name: str, package_version: str,
) -> list[str]:
return [f'{package_name}@{package_version}']


def node_get_additional_dependencies(
package_name: str, package_version: str,
) -> list[str]:
Expand All @@ -48,9 +64,11 @@ def rust_get_additional_dependencies(
'python': python_get_package_versions,
'ruby': ruby_get_package_versions,
'rust': rust_get_package_versions,
'golang': golang_get_package_versions,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

order

}

ADDITIONAL_DEPENDENCIES = {
'node': node_get_additional_dependencies,
'rust': rust_get_additional_dependencies,
'golang': golang_get_additional_dependencies,
}
2 changes: 1 addition & 1 deletion pre_commit_mirror_maker/make_repo.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def git(*cmd: str) -> None:
# Commit and tag
git('add', '.')
git('commit', '-m', f'Mirror: {version}')
git('tag', f'v{version}')
git('tag', f'v{version.lstrip("v")}')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm this seems not right



def make_repo(repo: str, *, language: str, name: str, **fmt_vars: str) -> None:
Expand Down
1 change: 1 addition & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ classifiers =
[options]
packages = find:
install_requires =
lxml
packaging
python_requires = >=3.7

Expand Down
7 changes: 7 additions & 0 deletions tests/languages_test.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from pre_commit_mirror_maker.languages import golang_get_package_versions
from pre_commit_mirror_maker.languages import node_get_package_versions
from pre_commit_mirror_maker.languages import python_get_package_versions
from pre_commit_mirror_maker.languages import ruby_get_package_versions
Expand Down Expand Up @@ -33,3 +34,9 @@ def test_rust_get_package_version_output():
ret = rust_get_package_versions('clap')
assert ret
assert_all_text(ret)


def test_golang_get_package_version_output():
ret = golang_get_package_versions('golang.org/x/tools/cmd/goimports')
assert ret
assert_all_text(ret)
22 changes: 22 additions & 0 deletions tests/make_repo_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -249,3 +249,25 @@ def test_rust_integration(in_git_dir):
assert _cmd('git', 'log', '--oneline')

# TODO: test that the package is installable


def test_golang_integration(in_git_dir):
make_repo(
'.',
language='golang', name='mvdan.cc/gofumpt', description='',
entry='gofumpt', id='gofumpt', match_key='types',
match_val='go', args='', require_serial='false',
minimum_pre_commit_version='3.0.0',
)
# Our files should exist
assert in_git_dir.join('.version').exists()
assert in_git_dir.join('.pre-commit-hooks.yaml').exists()
assert in_git_dir.join('go.mod').exists()
assert in_git_dir.join('main.go').exists()

# Should have made _some_ tags
assert _cmd('git', 'tag', '-l')
# Should have made _some_ commits
assert _cmd('git', 'log', '--oneline')

# TODO: test that the package is installable