Tests TestHandleGitHubAppWebHook and TestHandleGitHubWebHook are racey

[blkt]

Describe the issue

Said tests occasionally run into a race condition

      passthroughqueue.go:37: Passing message through queue: 931bc065-7fb5-4c44-bd0b-cd3bdeb5cb90
      passthroughqueue.go:37: Passing message through queue: 931bc065-7fb5-4c44-bd0b-cd3bdeb5cb90
      handlers_githubwebhooks_test.go:3330: 
          	Error Trace:	/home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3048
          	            				/home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3330
          	Error:      	Expected value not to be nil.
          	Test:       	TestAll/TestHandleGitHubAppWebHook/installation_repositories_added
          	Messages:   	no event received after waiting 10ms
  ==================
  WARNING: DATA RACE
  Write at 0x00c000822080 by goroutine 50:
    runtime.closechan()
        /opt/hostedtoolcache/go/1.23.3/x64/src/runtime/chan.go:397 +0x0
    github.com/mindersec/minder/internal/util/testqueue.(*PassthroughQueue).Close()
        /home/runner/work/minder/minder/internal/util/testqueue/passthroughqueue.go:44 +0xa4
    github.com/mindersec/minder/internal/providers/github/webhook.(*UnitTestSuite).TestHandleGitHubAppWebHook.func5.deferwrap3()
        /home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3265 +0x1f
    runtime.Goexit()
        /opt/hostedtoolcache/go/1.23.3/x64/src/runtime/panic.go:629 +0x5d
    testing.(*T).FailNow()
        <autogenerated>:1 +0x31
    github.com/stretchr/testify/require.NotNilf()
        /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/require/require.go:1595 +0x106
    github.com/mindersec/minder/internal/providers/github/webhook.(*UnitTestSuite).TestHandleGitHubAppWebHook.func3()
        /home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3048 +0x6a4
    github.com/mindersec/minder/internal/providers/github/webhook.(*UnitTestSuite).TestHandleGitHubAppWebHook.func5()
        /home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3330 +0x1c38
    testing.tRunner()
        /opt/hostedtoolcache/go/1.23.3/x64/src/testing/testing.go:1690 +0x226
    testing.(*T).Run.gowrap1()
        /opt/hostedtoolcache/go/1.23.3/x64/src/testing/testing.go:1743 +0x44
  
  Previous read at 0x00c000822080 by goroutine 399:
    runtime.chansend()
        /opt/hostedtoolcache/go/1.23.3/x64/src/runtime/chan.go:171 +0x0
    github.com/mindersec/minder/internal/util/testqueue.(*PassthroughQueue).Pass()
        /home/runner/work/minder/minder/internal/util/testqueue/passthroughqueue.go:38 +0x136
    github.com/mindersec/minder/internal/util/testqueue.(*PassthroughQueue).Pass-fm()
        <autogenerated>:1 +0x1f
    github.com/mindersec/minder/internal/events.(*eventer).Register.func1()
        /home/runner/work/minder/minder/internal/events/events.go:224 +0x93
    github.com/ThreeDotsLabs/watermill/message.(*Router).AddNoPublisherHandler.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router.go:350 +0x34
    github.com/ThreeDotsLabs/watermill/message/router/middleware.CorrelationID.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router/middleware/correlation.go:34 +0x49
    github.com/ThreeDotsLabs/watermill/message/router/middleware.Retry.Middleware-fm.Retry.Middleware.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router/middleware/retry.go:41 +0x78
    github.com/ThreeDotsLabs/watermill/message/router/middleware.poisonQueue.Middleware-fm.poisonQueue.Middleware.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router/middleware/poison.go:99 +0x141
    github.com/mindersec/minder/internal/events.NewEventer.recordMetrics.func3.1()
        /home/runner/work/minder/minder/internal/events/metrics.go:28 +0x159
    github.com/ThreeDotsLabs/watermill/components/metrics.HandlerPrometheusMetricsMiddleware.Middleware-fm.HandlerPrometheusMetricsMiddleware.Middleware.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/components/metrics/handler.go:58 +0x244
    github.com/ThreeDotsLabs/watermill/message.(*handler).handleMessage()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router.go:795 +0x28e
    github.com/ThreeDotsLabs/watermill/message.(*handler).run.gowrap2()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router.go:640 +0x4f
  
  Goroutine 50 (running) created at:
    testing.(*T).Run()
        /opt/hostedtoolcache/go/1.23.3/x64/src/testing/testing.go:1743 +0x825
    github.com/mindersec/minder/internal/providers/github/webhook.(*UnitTestSuite).TestHandleGitHubAppWebHook()
        /home/runner/work/minder/minder/internal/providers/github/webhook/handlers_githubwebhooks_test.go:3242 +0x64a4
    runtime.call16()
        /opt/hostedtoolcache/go/1.23.3/x64/src/runtime/asm_amd64.s:775 +0x42
    reflect.Value.Call()
        /opt/hostedtoolcache/go/1.23.3/x64/src/reflect/value.go:365 +0xb5
    github.com/stretchr/testify/suite.Run.func1()
        /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:[202](https://github.com/mindersec/minder/actions/runs/11816704502/job/32920560522?pr=4910#step:9:205) +0x6f1
    testing.tRunner()
        /opt/hostedtoolcache/go/1.23.3/x64/src/testing/testing.go:1690 +0x226
    testing.(*T).Run.gowrap1()
        /opt/hostedtoolcache/go/1.23.3/x64/src/testing/testing.go:1743 +0x44
  
  Goroutine 399 (running) created at:
    github.com/ThreeDotsLabs/watermill/message.(*handler).run()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router.go:640 +0x41e
    github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
        /home/runner/go/pkg/mod/github.com/!three!dots!labs/[email protected]/message/router.go:458 +0x1a4

To Reproduce

CI runs this (plus some additional options for coverage which are not relevant here)

go test -json -race -v ./...

I might be useful to add -count 100 or more.

What version are you using?

minder-0.20241113.3444_ref.84d517a