| title |
|---|
Gomote |
The gomote command is a client for the Go builder infrastructure. It's a remote control for remote Go builder machines.
$ go install golang.org/x/build/cmd/gomote@latest # Go 1.16 and later
The most basic usage of the gomote tool involves just a few steps:
- Create an instance.
- Push code to the instance.
- Run commands on the instance.
Running the create command will list available instance types.
$ gomote create
(list tons of buildlet types)
Then, an instance can be created by specifying an instance type. The instance's name will be printed to stdout, so the result may be stored in an environment variable. (There may be other logging messages, but they will be on stderr and each line will have a '#' prefix.)
$ gomote create linux-amd64
# still creating linux-amd64 (1) after 5s; 0 requests ahead of you
user-linux-amd64-0
With that instance's name you can now push (more specifically sync the contents of) a local Go repository to the instance and install a bootstrap toolchain. The repository you sync will appear at the go subdirectory of $WORKDIR (the default directory of all gomote operations). The bootstrap toolchain will always go into the go1.4 subdirectory (even if the bootstrap toolchain isn't from version 1.4).
$ GOROOT=/path/to/local/go/repo gomote push user-linux-amd64-0
$ gomote ls user-linux-amd64-0
go
go1.4
Note that push really is a "sync" operation, so next time you push the gomote tool will only push what has changed (files added, modified, or removed).
With a toolchain installed, you can now build it by running commands on the instance. The run command allows you to specify an executable to run. The executable must be specified relative to $WORKDIR (e.g. go/bin/go) or via an absolute path (e.g. /bin/bash). That executable will then run with its current working directory set to the directory containing the executable.
$ gomote run user-linux-amd64-0 go/src/make.bash
To then run the built Go toolchain, use go/bin/go.
$ gomote run user-linux-amd64-0 go/bin/go test -run="TestSomething" -v runtime
You can additionally specify a working directory and environment variables to run that will be applied before the command is executed.
Note that gomote instances will automatically disappear after 30 minutes of inactivity. Use the list command to check how long they have left.
$ gomote list
user-linux-amd64-0 linux-amd64 host-linux-amd64-bullseye expires in 10m27.339494527s
The ping command can be used to keep an instance alive if no other commands are being run against it.
For more information on each command run gomote help <command>. For more commands, run gomote help.
The create command contacts the build coordinator (farmer.golang.org) and requests that it create the buildlet on your behalf. All subsequent commands (such as gomote run or gomote ls) then proxy your request via the coordinator. To access a buildlet directly (for example, when working on the buildlet code), you can skip the gomote create step and use the special builder name <build-config-name>@ip[:port>, such as [email protected].
Instances may be managed in named groups, and commands are broadcast to all instances in the group.
A group is specified either by the -group global flag or through the GOMOTE_GROUP environment variable. The -group flag must always specify a valid group, whereas GOMOTE_GROUP may contain an invalid group. Instances may be part of more than one group.
Groups may be explicitly managed with the "group" subcommand, but there are several short-cuts that make this unnecessary in most cases:
- The
createcommand can create a new group for instances with the-new-groupflag. - The
createcommand will automatically create the group inGOMOTE_GROUPif it does not exist and no other group is explicitly specified. - The
destroycommand can destroy a group in addition to its instances with the-destroy-groupflag.
As a result, the easiest way to use groups is to just set the GOMOTE_GROUP environment variable:
$ export GOMOTE_GROUP=debug
$ gomote create linux-amd64
$ GOROOT=/path/to/goroot gomote push
$ gomote run go/src/make.bash
As this example demonstrates, groups are useful even if the group contains only a single instance: it can dramatically shorten most gomote commands.
The create command accepts the -setup flag which also pushes a GOROOT and runs the appropriate equivalent of make.bash for the instance.
Example:
$ GOROOT=/path/to/my/goroot gomote create -setup linux-amd64
# Creating user-linux-amd64-0...
# Pushing /path/to/my/goroot to user-linux-amd64-0
# Running make.bash on user-linux-amd64-0...
The create command accepts the -count flag for creating several instances at once.
Example:
$ gomote create -count=3 linux-amd64
# Creating user-linux-amd64-0...
# Creating user-linux-amd64-1...
# Creating user-linux-amd64-2...
The run command accepts the -collect flag for automatically writing the output from the command to a file in the current working directory, as well as a copy of the full file tree from the instance. This command is useful for capturing the output of long-running commands in a set-and-forget manner.
Example:
$ gomote run -collect user-linux-amd64-0 /bin/bash -c 'echo hi'
# Writing output to user-linux-amd64-0.stdout...
$ cat user-linux-amd64-0.stdout
hi
$ ls user-linux-amd64-0.tar.gz
user-linux-amd64-0.tar.gz
The run command accepts the -until flag for continuously executing a command until the output of the command matches some pattern. This is useful for reproducing rare issues, and especially useful when used together with -collect.
Example:
$ gomote run -collect -until 'FAIL' user-linux-amd64-0 go/bin/go test -run 'TestFlaky' -count=1000 runtime
# Writing output to user-linux-amd64-0.stdout...
$ cat user-linux-amd64-0.stdout
...
--- FAIL: TestFlaky ---
...
$ ls user-linux-amd64-0.tar.gz
user-linux-amd64-0.tar.gz
Note that the run command always streams output to a temporary file regardless of any additional flags to avoid losing output due to terminal scrollback. It always prints the location of the file.
Putting together some of the tricks above and making use of groups, it's much easier to hammer on some test to try to reproduce a rare failure. For example:
$ export GOMOTE_GROUP=debug
$ GOROOT=/path/to/goroot gomote create -setup -count=10 linux-amd64
$ gomote run -until='unexpected return pc' -collect go/bin/go run -run="TestFlaky" -count=100 runtime
Darwin gomotes hosted on LUCI do not have Xcode pre-installed. Without Xcode, they cannot do cgo builds. You can build with cgo disabled:
$ gomote run -e 'CGO_ENABLED=0' $MOTE go/src/make.bash
Or install Xcode like so:
$ gomote run $MOTE /bin/mkdir /tmp/xcode
$ gomote run $MOTE /Users/swarming/.swarming/w/ir/tools/bin/mac_toolchain install -xcode-version 15a240d -output-dir /tmp/xcode/Xcode.app
$ gomote run $MOTE /usr/bin/sudo xcode-select --switch /tmp/xcode/Xcode.app
Note: Depending on which machine you get, the mac_toolchain binary referenced may alternatively be at either /Volumes/Work/s/w/ir/tools/bin/mac_toolchain.
$ gomote run -path '$PATH,$WORKDIR/go/bin' $MOTE go/src/make.bat
$ gomote run -path '$PATH,$WORKDIR/go/bin' $MOTE go/bin/go.exe test cmd/go -short
Note: previous versions of the wiki have advised setting GOROOT for gomote 'run' commands (e.g. "-e GOROOT=c:\workdir\go"); this is no longer recommended (causes problems with Go command caching).
$ tar --exclude .git -C ~/go/src/ -zc golang.org/x/tools | gomote puttar -dir=gopath/src $MOTE -
$ gomote run -e 'GOPATH=c:/workdir/gopath' $MOTE go/bin/go test -run=TestFixImportsVendorPackage golang.org/x/tools/imports
If ssh'd into the machine, these envvars may be handy:
$ set GOPATH=c:\workdir\gopath
$ set PATH=%PATH%;c:\workdir\gopath\bin;c:\workdir\go\bin
$ set CGO_ENABLED=0
Testing golang.org/x/sys/unix on $MOTE
$ tar -C $GOPATH/src/ -zc golang.org/x/sys/unix | gomote puttar -dir=gopath/src $MOTE
$ gomote run -e 'GOPATH=/tmp/workdir/gopath' -dir 'gopath/src/golang.org/x/sys/unix' $MOTE go/bin/go test -v golang.org/x/sys/unix
(The GOPATH part is for GOPATH compatibility mode; the -dir is for modules mode, which looks in the working directory and up for go.mod)
export MOTE=`gomote create android-arm64-wikofever`
gomote push $MOTE
gomote run $MOTE go/src/make.bash
PATH must contain the exec wrapper, go_android_*_exec, built by make.bash.
gomote run -path '$PATH,$WORKDIR/go/bin' $MOTE go/bin/go test math/big
https://farmer.golang.org/builders lists information about how each buildlet is deployed and configured. The information is from golang.org/x/build/dashboard and golang.org/x/build/env.
On August 2022, a new infrastructure was deployed which required the removal of all gomote access from previously approved users. Please re-request access if you still require access.
To request access to the gomote service, file a new issue (https://go.dev/issue/new?title=access:+&body=See+https://go.dev/wiki/Gomote%23access.) and state the Google account you use to log in to Gerrit. The provided account will only be used for authentication purposes.
Authentication is triggered with the first invocation of a command:
$ gomote create linux-amd64
Please visit https://www.google.com/device in your browser and enter verification code:
ABCD-4567
...
After opening a browser with the provided link the user must authenticate with the google account and paste the verification code into the browser. After a short wait the client will be authenticated.
The gomote ssh command uses a SSH keys created specifically for gomote. On the first use of the gomote ssh a set of keys will be created and stored in the local user configuration directory. You may be asked to add set a password for the keys (a password is not required). The SSH functionality operates with OpenSSH certificate authentication and does not require any additional configuration. Not all builder types support gomote ssh.