hush: one-time secret links the server cannot read

Paste a secret, get a link, send it. The first person to open it and press
Reveal sees the secret; the link dies at that moment. The recipient needs a
browser and nothing else — no account, no client, no installed tooling.

The server cannot read what it stores. AES-256-GCM happens in the browser and
the key lives in the URL fragment, which browsers never transmit, so hushd
holds ciphertext and no key material. That is a property of where the key sits
rather than a promise about our conduct, which is why there is deliberately no
endpoint accepting a plaintext secret and no server-side-encryption fallback:
two guarantees behind one URL would be worse than one honest guarantee.

Three decisions carry the design:

  * GET /s/{id} touches NO storage, not even to check existence. Slack, Teams,
    WhatsApp, iMessage and Outlook Safe Links all fetch a URL before a human
    sees it, so destroying on GET would destroy most secrets in transit and the
    recipient's "already used" would be indistinguishable from interception.
    Only POST /reveal consumes. Bot user-agent detection is an arms race;
    removing the side effect from GET is not. Pinned by
    TestGettingTheRevealPageNeverConsumesTheSecret.
  * Destruction is one Redis GETDEL, which is atomic. GET-then-DEL has a window
    where two simultaneous readers both win, and for a one-time secret that
    window is the product. The store contract demands atomicity and the same
    concurrency test runs against both implementations.
  * Missing, already-revealed, expired and evicted are ONE indistinguishable
    410. Separating them would confirm to a prober that a given link was real.

The secret id IS the capability, so secret.ID is a struct whose every
accidental path — %v, %s, String(), slog, json.Marshal — emits a redacted
handle or refuses, and the raw value needs an explicit Value(). The first
version tried to prevent leaks by implementing no String() at all; its own test
caught that Go's fmt prints unexported fields anyway, so forbidding the method
had removed the control rather than the leak.

Operationally: structured JSON on stdout in the fleet's wire format, which
Vector already collects with no annotation; six hush_* metrics on the chassis
registry with no id, IP or path in any label; five alert rules wired into
vmalert. The public Ingress enumerates /, /s/ and /api/ so /metrics, /healthz
and /readyz share the port but are unreachable from the internet — no
basic-auth middleware to maintain and get wrong.

Dependencies are vendored because go-chassis is private: the Woodpecker test
step and the in-cluster Kaniko build both run -mod=vendor with GOPROXY=off and
hold no git credential.

cmd/hush-mcp is a stdio MCP server doing the same client-side crypto locally,
so using hush from an agent preserves the same guarantee as using it from a
browser.
This commit is contained in:
jx12n 2026-09-03 00:08:38 -06:00
commit 4d9a26498e
948 changed files with 369588 additions and 0 deletions

2
.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
.build/
*.tmp

62
.woodpecker.yml Normal file
View File

@ -0,0 +1,62 @@
# hush CI/CD. Origin must be Gitea (git.threesix.ai) — that remote carries the
# webhook. Pushing to the GitHub mirror does not deploy anything.
#
# push to main → test → Kaniko amd64 build → Zot → kubectl set image on `projects`
clone:
git:
image: woodpeckerci/plugin-git
settings:
depth: 1
steps:
test:
# 1.26, not the 1.25 in the template: go-chassis declares `go 1.26.0` and an
# older toolchain refuses to build it.
image: golang:1.26-alpine
environment:
# Hermetic, and deliberately so. go-chassis is a PRIVATE module and this
# container has no git credential, so a build that reached for the network
# would fail here — the vendored tree is what makes CI possible at all.
# GOPROXY=off turns "silently fetched from a proxy" into a hard error.
GOFLAGS: -mod=vendor
GOPROXY: "off"
commands:
- go vet ./...
- go test ./...
# gofmt as a gate, not a suggestion. -l lists offenders; the test fails if
# the list is non-empty. vendor/ is excluded because it is not ours.
- test -z "$(gofmt -l ./cmd ./internal)" || { gofmt -l ./cmd ./internal; echo "gofmt"; exit 1; }
when:
branch: main
event: push
build:
image: woodpeckerci/plugin-kaniko
settings:
registry: registry.threesix.ai
repo: hush/api
tags:
- latest
# SHA-tagged as well as latest: `latest` cannot pin a rollback.
- ${CI_COMMIT_SHA:0:8}
context: .
dockerfile: Dockerfile
cache: true
skip_tls_verify: true # internal Zot registry, self-signed cert
when:
branch: main
event: push
deploy:
image: bitnami/kubectl:latest
commands:
- kubectl set image deployment/hush hushd=registry.threesix.ai/hush/api:${CI_COMMIT_SHA:0:8} -n projects
- kubectl rollout status deployment/hush -n projects --timeout=120s
# Prove the rolled pod actually serves, not merely that it became Ready.
# A pod can pass readiness and still be the wrong image if `set image`
# silently matched nothing.
- kubectl -n projects get deployment hush -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
when:
branch: main
event: push

44
Dockerfile Normal file
View File

@ -0,0 +1,44 @@
# hushd. Built amd64 IN-CLUSTER by Kaniko — never `docker build` locally, which
# on an Apple Silicon laptop produces an arm64 image the cluster cannot run.
#
# The build is HERMETIC: `-mod=vendor` with `GOPROXY=off` and `GOFLAGS=-mod=vendor`
# means no module download and no network. That is not an optimisation — it is
# required, because github.com/orchard9/go-chassis is a private module and the
# build container holds no git credential. `make vendor` is what moves versions.
FROM golang:1.26-alpine AS build
WORKDIR /src
# Vendored, so this is the whole dependency graph: no go.sum verification step,
# no proxy, no cache warm-up.
COPY go.mod go.sum ./
COPY vendor/ ./vendor/
COPY cmd/ ./cmd/
COPY internal/ ./internal/
# Fail loudly if anything reaches for the network, rather than silently falling
# back to a proxy that will not be there in CI.
ENV GOFLAGS=-mod=vendor GOPROXY=off CGO_ENABLED=0
# Tests run in the CI step, not here: a Kaniko layer that runs tests caches
# their result and stops re-running them. Keep the image build to building.
RUN go build -trimpath -ldflags="-s -w" -o /out/hushd ./cmd/hushd
# distroless static + nonroot: no shell, no package manager, no libc surface.
# hushd needs only TCP and the CA bundle distroless already carries.
FROM gcr.io/distroless/static-debian12:nonroot AS runtime
WORKDIR /
COPY --from=build /out/hushd /hushd
# Numeric, not "nonroot": with a non-numeric USER the kubelet cannot verify
# runAsNonRoot and refuses to start the pod.
USER 65532:65532
# HTML templates are embedded in the binary (internal/web, go:embed), so there
# is no asset directory to mount, drift, or go missing at runtime.
EXPOSE 18500
# Exec form so SIGTERM reaches the process directly. The chassis's two-phase
# drain (readiness 503, wait, then shutdown) depends on receiving it, and the
# Deployment's terminationGracePeriodSeconds is sized to outlast that drain.
ENTRYPOINT ["/hushd"]

91
Makefile Normal file
View File

@ -0,0 +1,91 @@
.DEFAULT_GOAL := help
.PHONY: help fmt vet test test-redis build run dev dev-stop smoke vendor verify ci \
mcp mcp-install deploy-manifests deploy-status logs alerts-check
# Local development Redis. A real server, not a mock: the one-time guarantee
# rests on GETDEL being atomic, and a fake cannot prove that.
DEV_REDIS_PORT ?= 6389
DEV_REDIS_URL ?= redis://127.0.0.1:$(DEV_REDIS_PORT)/5
HUSH_PORT ?= 18500
BASE ?= http://127.0.0.1:$(HUSH_PORT)
# The public deployment. `hushd` runs one replica in the `projects` namespace.
KUBECONFIG_FILE ?= $(HOME)/.kube/orchard9-k3sf.yaml
NS ?= projects
HOST ?= hush.threesix.ai
help: ## Show this help
@grep -hE '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
| awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-18s\033[0m %s\n", $$1, $$2}'
fmt: ## Format everything outside vendor/
@gofmt -w ./cmd ./internal
vet: ## go vet
@go vet ./...
test: ## Unit tests. No Redis, no network, no container.
@go test ./...
test-redis: ## Run the store contract suite against a REAL Redis as well
@$(MAKE) --no-print-directory dev-redis
@HUSH_TEST_REDIS_URL=$(DEV_REDIS_URL) go test ./internal/store/... -count=1
build: ## Build both binaries into .build/
@mkdir -p .build
@go build -o .build/hushd ./cmd/hushd
@go build -o .build/hush-mcp ./cmd/hush-mcp
# `vendor` is not a convenience: github.com/orchard9/go-chassis is a PRIVATE
# module, so neither the Woodpecker test step nor the in-cluster Kaniko build
# can fetch it. Vendoring makes both hermetic — they build with -mod=vendor and
# no network and no credential. This target is the only way dep versions move.
vendor: ## Refresh vendor/ after a dependency change
@GOPRIVATE=github.com/orchard9 go mod tidy
@GOPRIVATE=github.com/orchard9 go mod vendor
@go build -mod=vendor ./... && echo "vendor/ builds hermetically"
verify: ## Prove the vendored tree builds with no network, exactly as CI does
@GOFLAGS=-mod=vendor GOPROXY=off go build ./... \
&& GOFLAGS=-mod=vendor GOPROXY=off go test ./... >/dev/null \
&& echo "hermetic build OK (GOPROXY=off, -mod=vendor)"
ci: fmt vet test verify ## Everything the pipeline runs
dev-redis:
@redis-cli -p $(DEV_REDIS_PORT) ping >/dev/null 2>&1 || { \
echo "starting redis on :$(DEV_REDIS_PORT)"; \
redis-server --port $(DEV_REDIS_PORT) --daemonize yes --save '' --appendonly no; \
for i in $$(seq 1 30); do redis-cli -p $(DEV_REDIS_PORT) ping >/dev/null 2>&1 && break; sleep 0.2; done; }
@redis-cli -p $(DEV_REDIS_PORT) ping | sed 's/^/redis: /'
dev: dev-redis build ## Run hushd locally against a local Redis
@echo "hushd on $(BASE) — open it in a browser"
@APP_ENV=dev REDIS_URL=$(DEV_REDIS_URL) HUSH_PORT=$(HUSH_PORT) .build/hushd
dev-stop: ## Stop the local Redis
@redis-cli -p $(DEV_REDIS_PORT) shutdown nosave 2>/dev/null || true
@echo "stopped"
smoke: ## Full create -> reveal -> gone against a locally running hushd
@BASE=$(BASE) ./scripts/smoke.sh
mcp: ## Build and install the MCP server, then register it with omp
@$(MAKE) --no-print-directory mcp-install
mcp-install:
@./scripts/install-mcp.sh
deploy-manifests: ## Apply the k8s manifests (do this BEFORE the first push)
@KUBECONFIG=$(KUBECONFIG_FILE) kubectl apply -f deployments/k8s/hush.yaml
deploy-status: ## Rollout, pods, ingress and certificate
@KUBECONFIG=$(KUBECONFIG_FILE) kubectl -n $(NS) rollout status deployment/hush --timeout=90s
@KUBECONFIG=$(KUBECONFIG_FILE) kubectl -n $(NS) get pod,svc,ingress -l app=hush
@KUBECONFIG=$(KUBECONFIG_FILE) kubectl -n $(NS) get certificate hush-tls 2>/dev/null || true
logs: ## Tail hush's structured logs out of VictoriaLogs
@./scripts/logs.sh
alerts-check: ## Confirm vmalert has loaded hush's rules
@./scripts/alerts-check.sh

118
README.md Normal file
View File

@ -0,0 +1,118 @@
# hush
Send someone a secret over a link that works once.
**Production:** <https://hush.threesix.ai>
Paste a secret, get a link, send the link. The first person to open it and press
**Reveal** sees the secret; the link is dead from that moment. Nobody needs an
account, a client, or anything installed — a browser is the whole requirement.
The server cannot read what you sent. Encryption happens in your browser and the
key lives in the URL *fragment* (`…/s/ID#KEY`), which browsers never transmit.
hush stores ciphertext it has no way to open. That is not a promise about our
operational discipline; it is a property of where the key sits.
## What one-time actually buys you
Worth being precise, because "one-time link" is often oversold:
- **Bounded exposure.** The secret is fetchable once, for at most its TTL, then it
is gone. A credential sitting in a Slack thread is fetchable forever by anyone
who later gains access to that thread.
- **Tamper evidence.** If your recipient says "already used", someone else opened
it. You have learned something a plain paste never tells you.
- **Nothing at rest to steal.** A dump of hush's Redis yields ciphertext and no keys.
And what it does not buy you:
- **It does not protect the link.** Whatever channel carries the link could be read
by whoever can read that channel. One-time-ness limits the damage and makes it
detectable; it does not make the channel private.
- **It does not authenticate the reader.** Anyone holding the link can open it.
The link *is* the capability. Treat it like the secret it carries.
If a secret must reach one specific verified human and nobody else, this is the
wrong tool — use a channel with identity.
## Usage
### In a browser
1. Open <https://hush.threesix.ai>.
2. Paste the secret, pick a lifetime, press **Create link**.
3. Copy the link and send it however you like.
4. The recipient opens it, presses **Reveal**, and reads it once.
### Why there is a button
Slack, Teams, WhatsApp, iMessage and Outlook Safe Links all fetch a URL to build
a preview *before* any human sees it. A service that destroys on `GET` therefore
destroys most secrets in transit, and the recipient's "already used" is
indistinguishable from a real interception.
So in hush, `GET /s/{id}` is a static page that touches no storage at all. Only
`POST /s/{id}/reveal` reads and destroys. Link previewers are harmless by
construction, not by user-agent guessing.
### API
The API takes **ciphertext**. There is no endpoint that accepts a plaintext
secret, because such an endpoint would make the server able to read secrets and
the claim at the top of this file would become a matter of trust rather than
arithmetic.
```
POST /api/secrets
{ "ciphertext": "<base64url AES-256-GCM, nonce prepended>", "ttl_seconds": 86400 }
→ 201 { "id": "…", "expires_at": "2026-09-04T…Z", "ttl_seconds": 86400 }
POST /api/secrets/{id}/reveal
→ 200 { "ciphertext": "…" } first caller only, secret destroyed
→ 410 { "error": { "code": "gone" } } every other case
```
`410 gone` is returned identically whether the id never existed, was already
revealed, or expired. Distinguishing those would confirm to an attacker that a
particular link once existed.
`GET /` serves the create page, `GET /s/{id}` the reveal page. `/healthz`,
`/readyz` and `/metrics` are served on the same port but are **not routed by the
public ingress** — they are reachable in-cluster only.
### From an agent, over MCP
`cmd/hush-mcp` is a stdio MCP server exposing two tools, `hush_create` and
`hush_reveal`. It runs **locally** and does the encryption on your machine, so
using hush from an agent preserves the same zero-knowledge property as using it
from a browser. See [docs/MCP.md](docs/MCP.md).
## Limits
| Thing | Value | Why |
|---|---|---|
| Ciphertext | ≤ 64 KiB | It is a courier for credentials, not a file host |
| TTL | 5m … 7d, default 24h | Long enough to be useful, short enough to bound exposure |
| Rate limit | 30 creates / 10 min / IP | Anonymous create is otherwise a free blob host |
| Reveals per secret | exactly 1 | The product |
## Operating it
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how it works and why each choice
- [docs/DEPLOY.md](docs/DEPLOY.md) — pipeline, DNS, credentials, first deploy
- [docs/OPERATIONS.md](docs/OPERATIONS.md) — alert runbook, log queries, failure modes
- [docs/MCP.md](docs/MCP.md) — the MCP server and how to install it
## Development
```bash
make help # every target
make test # unit tests, no external dependencies
make dev # a local Redis in Docker + hushd on :18500
make smoke # full create → reveal → gone against the local instance
make vendor # refresh vendor/ after a dependency change
```
`go-chassis` is a private module, so dependencies are **vendored** and both CI
and the container build run with `-mod=vendor` and no network. `make vendor` is
the only way dependency versions change.

141
cmd/hush-mcp/crypto_test.go Normal file
View File

@ -0,0 +1,141 @@
package main
import (
"encoding/base64"
"os/exec"
"strings"
"testing"
)
func TestSealOpenRoundTrip(t *testing.T) {
for _, plain := range []string{
"hunter2",
"",
strings.Repeat("x", 40000),
"unicode: ✓ 漢字 🔐",
"multi\nline\nwith\ttabs",
} {
ct, key, err := seal(plain)
if err != nil {
t.Fatalf("seal(%d bytes): %v", len(plain), err)
}
got, err := open(ct, key)
if err != nil {
t.Fatalf("open: %v", err)
}
if got != plain {
t.Fatalf("round trip changed the plaintext (%d bytes)", len(plain))
}
}
}
func TestSealProducesTheBrowsersWireFormat(t *testing.T) {
ct, key, err := seal("x")
if err != nil {
t.Fatal(err)
}
// base64url, unpadded, matching base64.RawURLEncoding on the server and
// the b64u helper in templates/base.html. Three implementations, one
// spelling — a mismatch here means a link minted by one client cannot be
// opened by another.
for name, v := range map[string]string{"ciphertext": ct, "key": key} {
if strings.ContainsAny(v, "+/=") {
t.Fatalf("%s %q uses the standard base64 alphabet or padding; it must be base64url unpadded", name, v)
}
if _, err := base64.RawURLEncoding.DecodeString(v); err != nil {
t.Fatalf("%s does not decode as base64url: %v", name, err)
}
}
// 256-bit key.
raw, _ := base64.RawURLEncoding.DecodeString(key)
if len(raw) != 32 {
t.Fatalf("key is %d bytes, want 32 (AES-256)", len(raw))
}
// 96-bit nonce prepended, then at least the GCM tag.
blob, _ := base64.RawURLEncoding.DecodeString(ct)
if len(blob) < 12+16 {
t.Fatalf("ciphertext is %d bytes, too short for a 12-byte nonce plus a 16-byte tag", len(blob))
}
}
func TestOpenRefusesAWrongKeyAndTamperedCiphertext(t *testing.T) {
ct, _, err := seal("secret")
if err != nil {
t.Fatal(err)
}
_, otherKey, _ := seal("unrelated")
if _, err := open(ct, otherKey); err == nil {
t.Fatal("open() accepted a key that does not belong to this ciphertext")
}
// GCM is authenticated: a flipped byte must fail, not decrypt to garbage.
blob, _ := base64.RawURLEncoding.DecodeString(ct)
blob[len(blob)-1] ^= 0xff
_, key, _ := seal("x")
if _, err := open(base64.RawURLEncoding.EncodeToString(blob), key); err == nil {
t.Fatal("open() accepted tampered ciphertext")
}
if _, err := open("!!!not base64!!!", key); err == nil {
t.Fatal("open() accepted a non-base64url ciphertext")
}
if _, err := open(ct, "!!!"); err == nil {
t.Fatal("open() accepted a non-base64url key")
}
}
// Cross-implementation check: a blob sealed by this Go client must decrypt with
// Python's AES-GCM, and vice versa. This is what proves the MCP client, the
// browser and the smoke script really share one format rather than three
// self-consistent ones.
func TestWireFormatMatchesAnIndependentImplementation(t *testing.T) {
if _, err := exec.LookPath("python3"); err != nil {
t.Skip("python3 unavailable")
}
const plain = "cross-implementation-secret"
ct, key, err := seal(plain)
if err != nil {
t.Fatal(err)
}
// Go -> Python
out, err := exec.Command("python3", "-c", `
import base64, sys
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
u = lambda s: base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
blob, key = u(sys.argv[1]), u(sys.argv[2])
sys.stdout.write(AESGCM(key).decrypt(blob[:12], blob[12:], None).decode())
`, ct, key).Output()
if err != nil {
t.Skipf("python cryptography unavailable: %v", err)
}
if string(out) != plain {
t.Fatalf("python decrypted our ciphertext to %q, want %q", out, plain)
}
// Python -> Go
pyOut, err := exec.Command("python3", "-c", `
import base64, os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
key = AESGCM.generate_key(bit_length=256); nonce = os.urandom(12)
blob = nonce + AESGCM(key).encrypt(nonce, b"from-python", None)
b = lambda x: base64.urlsafe_b64encode(x).decode().rstrip("=")
print(b(blob), b(key))
`).Output()
if err != nil {
t.Fatalf("python encrypt failed: %v", err)
}
parts := strings.Fields(string(pyOut))
if len(parts) != 2 {
t.Fatalf("unexpected python output %q", pyOut)
}
got, err := open(parts[0], parts[1])
if err != nil {
t.Fatalf("could not open python's ciphertext: %v", err)
}
if got != "from-python" {
t.Fatalf("opened python's ciphertext to %q", got)
}
}

304
cmd/hush-mcp/main.go Normal file
View File

@ -0,0 +1,304 @@
// Command hush-mcp exposes hush to an MCP host (omp, Claude Code, any client)
// as two tools: hush_create and hush_reveal.
//
// It runs LOCALLY, on the operator's machine, and does the AES-256-GCM itself.
// That is the whole reason it is a stdio binary rather than an endpoint served
// by hushd: if the server did the encrypting, the server could read every
// secret created through MCP, and hush's central claim would hold for browser
// users and quietly not hold for agent users. Two guarantees behind one URL is
// worse than one honest guarantee.
//
// So this binary is a peer of the browser, not a peer of the server: it mints
// the key, encrypts, posts ciphertext, and assembles the `#fragment` link.
// hushd sees exactly what it sees from a browser.
package main
import (
"bytes"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strings"
"time"
)
const version = "0.1.0"
// stderr is indirected so tests can capture diagnostics. stdout is reserved
// for protocol frames — a stray write there corrupts the stream.
var stderr io.Writer = os.Stderr
// b64 matches the server and the browser: base64url, unpadded. One spelling of
// the wire format across all three implementations.
var b64 = base64.RawURLEncoding
func main() {
base := strings.TrimSuffix(os.Getenv("HUSH_BASE_URL"), "/")
if base == "" {
base = "https://hush.threesix.ai"
}
c := &client{
base: base,
http: &http.Client{Timeout: 15 * time.Second},
// A create token is only needed if the deployment has closed anonymous
// create (HUSH_REQUIRE_AUTH). Empty is the normal case.
token: os.Getenv("HUSH_CREATE_TOKEN"),
}
srv := NewServer("hush", version, os.Stdout, tools(c))
if err := srv.Serve(os.Stdin); err != nil {
fmt.Fprintf(stderr, "hush-mcp: %v\n", err)
os.Exit(1)
}
}
func tools(c *client) []Tool {
return []Tool{
{
Name: "hush_create",
Title: "Create a one-time secret link",
Description: "Encrypt a secret locally and store the ciphertext on hush, returning a " +
"link that works exactly once. The encryption key is generated on this machine and " +
"travels only in the link's #fragment, so the hush server never receives it and " +
"cannot read the secret. Use this to hand a credential to someone instead of " +
"pasting it into chat or email. The returned link is shown once and cannot be " +
"recovered — pass it on immediately.",
Schema: map[string]any{
"type": "object",
"properties": map[string]any{
"secret": map[string]any{
"type": "string",
"description": "The plaintext to send. Never leaves this machine unencrypted.",
},
"ttl_seconds": map[string]any{
"type": "integer",
"description": "Lifetime in seconds, 300 to 604800. Defaults to 86400 (24h).",
"minimum": 300,
"maximum": 604800,
},
},
"required": []string{"secret"},
"additionalProperties": false,
},
Handler: c.create,
},
{
Name: "hush_reveal",
Title: "Open a one-time secret link",
Description: "Fetch and decrypt a hush link, DESTROYING it in the process. This is " +
"irreversible: after this call the link is dead and nobody else can open it, " +
"including the intended recipient. Only call this on a link meant for you.",
Schema: map[string]any{
"type": "object",
"properties": map[string]any{
"link": map[string]any{
"type": "string",
"description": "The full hush link including the #fragment key.",
},
},
"required": []string{"link"},
"additionalProperties": false,
},
Handler: c.reveal,
},
}
}
type client struct {
base string
http *http.Client
token string
}
func (c *client) create(raw json.RawMessage) (string, error) {
var args struct {
Secret string `json:"secret"`
TTLSeconds int64 `json:"ttl_seconds"`
}
if err := json.Unmarshal(raw, &args); err != nil {
return "", fmt.Errorf("bad arguments: %w", err)
}
if args.Secret == "" {
return "", errors.New("secret is empty — nothing to send")
}
ciphertext, key, err := seal(args.Secret)
if err != nil {
return "", fmt.Errorf("encrypt locally: %w", err)
}
body, _ := json.Marshal(map[string]any{"ciphertext": ciphertext, "ttl_seconds": args.TTLSeconds})
req, err := http.NewRequest(http.MethodPost, c.base+"/api/secrets", bytes.NewReader(body))
if err != nil {
return "", err
}
req.Header.Set("Content-Type", "application/json")
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
res, err := c.http.Do(req)
if err != nil {
return "", fmt.Errorf("reach hush at %s: %w", c.base, err)
}
defer res.Body.Close()
payload, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
if res.StatusCode != http.StatusCreated {
return "", fmt.Errorf("hush refused the secret (%s): %s", res.Status, apiMessage(payload))
}
var out struct {
ID string `json:"id"`
ExpiresAt string `json:"expires_at"`
TTLSeconds int64 `json:"ttl_seconds"`
}
if err := json.Unmarshal(payload, &out); err != nil || out.ID == "" {
return "", fmt.Errorf("unexpected response from hush: %s", string(payload))
}
// The key is appended HERE, on this machine. It was never in the request.
link := c.base + "/s/" + out.ID + "#" + key
return fmt.Sprintf(
"%s\n\nOne-time link — works exactly once, expires %s.\n"+
"The key is in the #fragment, so hush cannot read the secret.\n"+
"This link cannot be shown again: send it now.",
link, out.ExpiresAt), nil
}
func (c *client) reveal(raw json.RawMessage) (string, error) {
var args struct {
Link string `json:"link"`
}
if err := json.Unmarshal(raw, &args); err != nil {
return "", fmt.Errorf("bad arguments: %w", err)
}
u, err := url.Parse(strings.TrimSpace(args.Link))
if err != nil {
return "", fmt.Errorf("not a URL: %w", err)
}
if u.Fragment == "" {
// The commonest real failure: a chat client or mail rewriter dropped
// the fragment. Say so precisely, because the secret is still intact
// and the fix is to ask the sender for the whole link.
return "", errors.New("this link has no #fragment, so it carries no key. " +
"Chat and email clients sometimes truncate it — ask the sender for the full link. " +
"The secret has NOT been opened.")
}
id := strings.TrimPrefix(u.Path, "/s/")
if id == "" || strings.Contains(id, "/") {
return "", fmt.Errorf("cannot find a secret id in the path %q", u.Path)
}
// Reveal against the link's OWN origin, not the configured base: a link
// from a different hush deployment must not be posted to this one, where
// the id would be meaningless.
origin := u.Scheme + "://" + u.Host
res, err := c.http.Post(origin+"/api/secrets/"+url.PathEscape(id)+"/reveal", "application/json", nil)
if err != nil {
return "", fmt.Errorf("reach hush at %s: %w", origin, err)
}
defer res.Body.Close()
payload, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
if res.StatusCode == http.StatusGone {
return "", errors.New("gone: this link was already opened, expired, or never existed. " +
"If you did not open it yourself, assume someone else did and ask the sender to rotate the secret.")
}
if res.StatusCode != http.StatusOK {
return "", fmt.Errorf("hush returned %s: %s", res.Status, apiMessage(payload))
}
var out struct {
Ciphertext string `json:"ciphertext"`
}
if err := json.Unmarshal(payload, &out); err != nil {
return "", fmt.Errorf("unexpected response from hush: %s", string(payload))
}
plain, err := open(out.Ciphertext, u.Fragment)
if err != nil {
// The ciphertext is already destroyed at this point, so there is
// nothing to retry. Say that plainly.
return "", fmt.Errorf("the key in this link does not open this secret, and the ciphertext "+
"has now been destroyed (%w). The link was probably altered in transit; ask for a new one", err)
}
return plain, nil
}
// seal encrypts with AES-256-GCM and returns (ciphertext, key), both base64url.
// The nonce is prepended to the ciphertext so the stored blob is self-contained
// — byte-for-byte the format templates/base.html produces.
func seal(plaintext string) (string, string, error) {
key := make([]byte, 32)
if _, err := rand.Read(key); err != nil {
return "", "", err
}
block, err := aes.NewCipher(key)
if err != nil {
return "", "", err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return "", "", err
}
nonce := make([]byte, gcm.NonceSize())
if _, err := rand.Read(nonce); err != nil {
return "", "", err
}
blob := gcm.Seal(nonce, nonce, []byte(plaintext), nil)
return b64.EncodeToString(blob), b64.EncodeToString(key), nil
}
func open(ciphertext, keyStr string) (string, error) {
blob, err := b64.DecodeString(ciphertext)
if err != nil {
return "", fmt.Errorf("ciphertext is not base64url: %w", err)
}
key, err := b64.DecodeString(keyStr)
if err != nil {
return "", fmt.Errorf("key is not base64url: %w", err)
}
block, err := aes.NewCipher(key)
if err != nil {
return "", err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return "", err
}
if len(blob) < gcm.NonceSize() {
return "", errors.New("ciphertext is too short to contain a nonce")
}
plain, err := gcm.Open(nil, blob[:gcm.NonceSize()], blob[gcm.NonceSize():], nil)
if err != nil {
return "", errors.New("authentication failed")
}
return string(plain), nil
}
// apiMessage pulls the human message out of hush's error envelope, falling back
// to the raw body so a proxy's HTML error page is still readable.
func apiMessage(payload []byte) string {
var e struct {
Error struct {
Code string `json:"code"`
Message string `json:"message"`
} `json:"error"`
}
if err := json.Unmarshal(payload, &e); err == nil && e.Error.Message != "" {
return e.Error.Code + ": " + e.Error.Message
}
s := strings.TrimSpace(string(payload))
if len(s) > 300 {
s = s[:300] + "…"
}
return s
}

195
cmd/hush-mcp/mcp.go Normal file
View File

@ -0,0 +1,195 @@
package main
import (
"bufio"
"encoding/json"
"fmt"
"io"
"sync"
)
// A minimal, dependency-free MCP server over stdio.
//
// MCP on stdio is newline-delimited JSON-RPC 2.0. The surface this server needs
// is four methods — initialize, notifications/initialized, tools/list,
// tools/call — so it is implemented directly rather than pulling in an SDK
// whose API churn would be a bigger maintenance surface than the protocol.
//
// The one rule that matters for a stdio server: stdout carries protocol frames
// ONLY. Anything diagnostic goes to stderr, because a stray Println on stdout
// corrupts the stream and the host reports an opaque parse failure.
// protocolVersion is the MCP revision this server implements. The host sends
// its own in initialize; the spec has the server answer with the version it
// will actually speak rather than echoing the client's.
const protocolVersion = "2025-06-18"
type request struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id,omitempty"`
Method string `json:"method"`
Params json.RawMessage `json:"params,omitempty"`
}
type response struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id,omitempty"`
Result any `json:"result,omitempty"`
Error *rpcError `json:"error,omitempty"`
}
type rpcError struct {
Code int `json:"code"`
Message string `json:"message"`
}
// JSON-RPC 2.0 reserved codes. -32602 is the one that matters here: a bad tool
// argument is an invalid-params error, not a transport failure.
const (
codeInvalidParams = -32602
codeMethodMissing = -32601
codeInternal = -32603
)
// Tool is one callable tool. Schema is the raw JSON Schema advertised to the
// host, which is what makes the arguments self-documenting in the client.
type Tool struct {
Name string
Title string
Description string
Schema map[string]any
// Handler returns the text to show the caller. An error is reported as a
// TOOL error (isError on the result) rather than a protocol error, so the
// model sees the message and can act on it instead of the call appearing
// to have failed at the transport level.
Handler func(args json.RawMessage) (string, error)
}
// Server dispatches MCP over a reader/writer pair.
type Server struct {
name string
version string
tools []Tool
mu sync.Mutex // serialises writes: one frame per line, never interleaved
out *json.Encoder
w io.Writer
}
func NewServer(name, version string, out io.Writer, tools []Tool) *Server {
return &Server{name: name, version: version, tools: tools, out: json.NewEncoder(out), w: out}
}
// Serve reads frames until stdin closes, which is how a stdio host signals
// shutdown.
func (s *Server) Serve(in io.Reader) error {
sc := bufio.NewScanner(in)
// A tool result can carry a secret link, which is small, but the buffer is
// raised so a large argument cannot truncate a frame into a parse error.
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
for sc.Scan() {
line := sc.Bytes()
if len(line) == 0 {
continue
}
var req request
if err := json.Unmarshal(line, &req); err != nil {
// Malformed frame with no id: nothing to reply to. Report on stderr
// and keep the stream alive.
fmt.Fprintf(stderr, "hush-mcp: unparseable frame: %v\n", err)
continue
}
s.dispatch(req)
}
return sc.Err()
}
func (s *Server) dispatch(req request) {
// A notification has no id and MUST NOT be answered. Replying to one is the
// classic stdio bug: the host sees an unsolicited response and desyncs.
isNotification := len(req.ID) == 0
switch req.Method {
case "initialize":
s.reply(req.ID, map[string]any{
"protocolVersion": protocolVersion,
"capabilities": map[string]any{"tools": map[string]any{}},
"serverInfo": map[string]any{"name": s.name, "version": s.version},
})
case "notifications/initialized":
// Handshake complete. Nothing to send.
case "ping":
s.reply(req.ID, map[string]any{})
case "tools/list":
list := make([]map[string]any, 0, len(s.tools))
for _, t := range s.tools {
list = append(list, map[string]any{
"name": t.Name,
"title": t.Title,
"description": t.Description,
"inputSchema": t.Schema,
})
}
s.reply(req.ID, map[string]any{"tools": list})
case "tools/call":
s.call(req)
default:
if !isNotification {
s.fail(req.ID, codeMethodMissing, "unsupported method: "+req.Method)
}
}
}
func (s *Server) call(req request) {
var p struct {
Name string `json:"name"`
Arguments json.RawMessage `json:"arguments"`
}
if err := json.Unmarshal(req.Params, &p); err != nil {
s.fail(req.ID, codeInvalidParams, "malformed tools/call params")
return
}
for _, t := range s.tools {
if t.Name != p.Name {
continue
}
text, err := t.Handler(p.Arguments)
if err != nil {
// isError:true keeps this a TOOL failure the model can read and
// react to, rather than a protocol error that looks like the server
// broke.
s.reply(req.ID, map[string]any{
"content": []map[string]any{{"type": "text", "text": err.Error()}},
"isError": true,
})
return
}
s.reply(req.ID, map[string]any{
"content": []map[string]any{{"type": "text", "text": text}},
})
return
}
s.fail(req.ID, codeInvalidParams, "unknown tool: "+p.Name)
}
func (s *Server) reply(id json.RawMessage, result any) {
if len(id) == 0 {
return
}
s.write(response{JSONRPC: "2.0", ID: id, Result: result})
}
func (s *Server) fail(id json.RawMessage, code int, msg string) {
if len(id) == 0 {
return
}
s.write(response{JSONRPC: "2.0", ID: id, Error: &rpcError{Code: code, Message: msg}})
}
func (s *Server) write(r response) {
s.mu.Lock()
defer s.mu.Unlock()
if err := s.out.Encode(r); err != nil {
fmt.Fprintf(stderr, "hush-mcp: write failed: %v\n", err)
}
}

65
cmd/hushd/config.go Normal file
View File

@ -0,0 +1,65 @@
package main
import (
"time"
"github.com/orchard9/go-chassis/config"
)
// Config is every knob hushd has. Loaded once at boot; a missing or malformed
// required value stops the process rather than being defaulted, so a
// misconfigured pod crashloops visibly instead of serving something subtly
// wrong.
type Config struct {
Env string
Port int
// RedisURL is REQUIRED. There is an in-memory store in the codebase for
// tests, and requiring this is what makes it impossible to select by
// accident in production: a pod with no REDIS_URL does not boot, rather
// than booting with a store that loses every secret on restart.
RedisURL string
// RateLimit bounds anonymous creates per client IP. Reveals are not
// limited by this: a recipient gets exactly one successful reveal by
// construction, so there is nothing to throttle, and throttling would let
// one noisy NAT block a colleague's delivery.
RateLimitCreates int
RateLimitWindow time.Duration
// TrustedProxyHops is how many Traefik hops sit in front of hushd, used to
// pick the real client IP out of X-Forwarded-For. Wrong-high lets a caller
// spoof their IP and evade the rate limit; wrong-low rate-limits the
// ingress itself and throttles everyone together.
TrustedProxyHops int
// AllowOrigins is the CORS allowlist. Empty is correct for the deployed
// service: both pages are same-origin, so no cross-origin caller is
// legitimate.
AllowOrigins []string
// RequireAuthToCreate closes anonymous create if the service is abused.
// Reveal stays anonymous regardless — the recipient is external and holds
// no credential.
RequireAuthToCreate bool
CreateToken string
}
func loadConfig() (Config, error) {
l := config.New()
cfg := Config{
Env: l.OneOf("APP_ENV", "dev", "dev", "staging", "prod"),
Port: l.Port("HUSH_PORT", 18500),
RedisURL: l.Required("REDIS_URL"),
RateLimitCreates: l.Int("HUSH_RATE_LIMIT_CREATES", 30),
RateLimitWindow: l.Duration("HUSH_RATE_LIMIT_WINDOW", 10*time.Minute),
TrustedProxyHops: l.Int("HUSH_TRUSTED_PROXY_HOPS", 1),
AllowOrigins: l.Strings("HUSH_ALLOW_ORIGINS", nil),
RequireAuthToCreate: l.Bool("HUSH_REQUIRE_AUTH", false),
CreateToken: l.String("HUSH_CREATE_TOKEN", ""),
}
if err := l.Err(); err != nil {
return Config{}, err
}
return cfg, nil
}

183
cmd/hushd/handlers.go Normal file
View File

@ -0,0 +1,183 @@
package main
import (
"errors"
"net/http"
"time"
"github.com/orchard9/go-chassis/chassis"
"github.com/orchard9/hush/internal/secret"
"github.com/orchard9/hush/internal/store"
"github.com/orchard9/hush/internal/web"
)
// Server holds the handler dependencies.
type Server struct {
cfg Config
store store.Store
pages *web.Pages
metrics *Metrics
}
// pageData is the same for every render: the limits, so the browser enforces
// what the server enforces. It carries nothing request-specific, which is why
// both pages are safely static.
func (s *Server) pageData() web.Data {
return web.Data{
MaxCiphertextBytes: secret.MaxCiphertextBytes,
DefaultTTLSeconds: int(secret.DefaultTTL.Seconds()),
MinTTLSeconds: int(secret.MinTTL.Seconds()),
MaxTTLSeconds: int(secret.MaxTTL.Seconds()),
}
}
// handleCreatePage serves GET /.
func (s *Server) handleCreatePage(c *chassis.Context) error {
return s.pages.Create(c.Writer(), s.pageData())
}
// handleRevealPage serves GET /s/{id}.
//
// It touches NO storage — not even to check whether the id exists. That is the
// design's load-bearing property: Slack, Teams, WhatsApp, iMessage and Outlook
// Safe Links all fetch URLs before a human sees them, so any read here would
// destroy most secrets in transit. It also means this response cannot leak
// whether an id exists.
//
// The id is not even parsed: a malformed one gets the same page, and the POST
// is what answers. Validating here would make this endpoint an id oracle.
func (s *Server) handleRevealPage(c *chassis.Context) error {
return s.pages.Reveal(c.Writer(), s.pageData())
}
type createRequest struct {
Ciphertext string `json:"ciphertext"`
TTLSeconds int64 `json:"ttl_seconds"`
}
type createResponse struct {
ID string `json:"id"`
ExpiresAt string `json:"expires_at"`
TTLSeconds int64 `json:"ttl_seconds"`
}
// handleCreate serves POST /api/secrets.
//
// It accepts ciphertext only. There is deliberately no endpoint taking a
// plaintext secret: one would make the server able to read secrets, and then
// nobody could tell from a link which guarantee they had.
func (s *Server) handleCreate(c *chassis.Context) error {
var req createRequest
if err := c.Bind(&req); err != nil {
s.metrics.Rejected.WithLabelValues(reasonMalformed).Inc()
return err
}
if err := secret.ValidateCiphertext(req.Ciphertext); err != nil {
reason, code := classifyCiphertextError(err)
s.metrics.Rejected.WithLabelValues(reason).Inc()
// err carries sizes, never content: ValidateCiphertext never puts the
// ciphertext in its message.
return &chassis.Error{Status: http.StatusUnprocessableEntity, Code: code, Msg: err.Error()}
}
ttl, err := secret.ResolveTTL(time.Duration(req.TTLSeconds) * time.Second)
if err != nil {
s.metrics.Rejected.WithLabelValues(reasonTTL).Inc()
return &chassis.Error{Status: http.StatusUnprocessableEntity, Code: reasonTTL, Msg: err.Error()}
}
id, err := secret.NewID()
if err != nil {
// No entropy means no safe id. Refuse rather than mint a guessable one.
return chassis.Internal(err)
}
if err := s.store.Put(c.Context(), id, req.Ciphertext, ttl); err != nil {
if errors.Is(err, store.ErrIDCollision) {
// Impossible at 256 bits, so it means an id-generation bug. Surfaced
// rather than retried, because a retry loop would hide it.
c.Log().Error("secret.id_collision", "category", "secret",
"error_type", "id_collision", "sid", id.LogHandle())
return chassis.Internal(err)
}
return chassis.Internal(err)
}
s.metrics.Created.Inc()
s.metrics.Bytes.Observe(float64(len(req.Ciphertext)))
// sid, not id: the id is the capability and never reaches a log.
c.Log().Info("secret.created", "category", "secret",
"sid", id.LogHandle(), "ttl_seconds", int64(ttl.Seconds()),
"ciphertext_bytes", len(req.Ciphertext))
return c.Created(createResponse{
ID: id.Value(),
ExpiresAt: time.Now().UTC().Add(ttl).Format(time.RFC3339),
TTLSeconds: int64(ttl.Seconds()),
})
}
type revealResponse struct {
Ciphertext string `json:"ciphertext"`
}
// gone is the single response for every unavailable secret: never existed,
// already revealed, expired, or evicted early by Redis.
//
// One response for four causes is deliberate. A caller able to tell "already
// revealed" from "never existed" learns that a particular link was real, which
// is information about someone else's secret.
func gone() *chassis.Error {
return &chassis.Error{
Status: http.StatusGone,
Code: "gone",
Msg: "this secret is not available: it was already revealed, expired, or never existed",
}
}
// handleReveal serves POST /api/secrets/{id}/reveal — the only destructive
// route, and the reason GET is inert.
func (s *Server) handleReveal(c *chassis.Context) error {
id, err := secret.ParseID(c.PathValue("id"))
if err != nil {
// A malformed id gets the SAME 410 as a missing one. A 400 here would
// separate "not an id we could have minted" from "an id that is gone",
// which is a free oracle for anyone probing the id space.
s.metrics.Revealed.WithLabelValues("gone").Inc()
return gone()
}
ciphertext, err := s.store.Take(c.Context(), id)
switch {
case errors.Is(err, store.ErrGone):
s.metrics.Revealed.WithLabelValues("gone").Inc()
c.Log().Info("secret.gone", "category", "secret", "sid", id.LogHandle())
return gone()
case err != nil:
// A store error is NOT reported as gone. The secret may still exist,
// and telling the recipient it is gone would send them to rotate a
// credential that was never delivered.
return chassis.Internal(err)
}
s.metrics.Revealed.WithLabelValues("ok").Inc()
c.Log().Info("secret.revealed", "category", "secret", "sid", id.LogHandle())
return c.OK(revealResponse{Ciphertext: ciphertext})
}
// classifyCiphertextError maps a validation failure to its metric reason and
// API error code, which are the same vocabulary on purpose.
func classifyCiphertextError(err error) (reason, code string) {
switch {
case errors.Is(err, secret.ErrCiphertextTooLarge):
return reasonTooLarge, reasonTooLarge
case errors.Is(err, secret.ErrCiphertextInvalid):
return reasonInvalid, reasonInvalid
case errors.Is(err, secret.ErrCiphertextEmpty):
return reasonEmpty, reasonEmpty
default:
return reasonMalformed, reasonMalformed
}
}

294
cmd/hushd/handlers_test.go Normal file
View File

@ -0,0 +1,294 @@
package main
import (
"encoding/base64"
"encoding/json"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/orchard9/go-chassis/chassis"
"github.com/orchard9/hush/internal/secret"
"github.com/orchard9/hush/internal/store"
"github.com/orchard9/hush/internal/web"
)
// testApp builds the real route table over an in-memory store, so these tests
// exercise the actual middleware chain, binder and error envelopes rather than
// calling handlers directly.
func testApp(t *testing.T) (http.Handler, *store.Memory) {
t.Helper()
pages, err := web.New()
if err != nil {
t.Fatal(err)
}
mem := store.NewMemory()
srv := &Server{
cfg: Config{Env: "dev"},
store: mem,
pages: pages,
metrics: newMetrics(),
}
srv.metrics.Prime()
log := slog.New(slog.NewJSONHandler(io.Discard, nil))
app := chassis.New(chassis.Config{Service: "hush", Env: "dev", MaxBodyBytes: 128 * 1024}, log)
app.Get("/", srv.handleCreatePage)
app.Get("/s/{id}", srv.handleRevealPage)
app.Route("/api", func(r *chassis.Router) {
r.Post("/secrets", srv.handleCreate)
r.Post("/secrets/{id}/reveal", srv.handleReveal)
})
return app.Handler(), mem
}
func ciphertext(s string) string {
return base64.RawURLEncoding.EncodeToString([]byte(s))
}
func do(t *testing.T, h http.Handler, method, path, body string) (int, map[string]any, string) {
t.Helper()
var r *http.Request
if body == "" {
r = httptest.NewRequest(method, path, nil)
} else {
r = httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Content-Type", "application/json")
}
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
raw := w.Body.String()
var parsed map[string]any
_ = json.Unmarshal([]byte(raw), &parsed)
return w.Code, parsed, raw
}
func create(t *testing.T, h http.Handler, ct string) string {
t.Helper()
code, body, raw := do(t, h, http.MethodPost, "/api/secrets",
`{"ciphertext":"`+ct+`","ttl_seconds":3600}`)
if code != http.StatusCreated {
t.Fatalf("create = %d, want 201: %s", code, raw)
}
id, ok := body["id"].(string)
if !ok || id == "" {
t.Fatalf("create response carried no id: %s", raw)
}
return id
}
func TestCreateThenRevealThenGone(t *testing.T) {
h, _ := testApp(t)
ct := ciphertext("the-secret-bytes")
id := create(t, h, ct)
code, body, raw := do(t, h, http.MethodPost, "/api/secrets/"+id+"/reveal", "")
if code != http.StatusOK {
t.Fatalf("first reveal = %d, want 200: %s", code, raw)
}
if body["ciphertext"] != ct {
t.Fatalf("reveal returned %v, want the stored ciphertext", body["ciphertext"])
}
code, _, raw = do(t, h, http.MethodPost, "/api/secrets/"+id+"/reveal", "")
if code != http.StatusGone {
t.Fatalf("second reveal = %d, want 410: %s", code, raw)
}
}
// THE regression test for this whole design. Slack, Teams, WhatsApp, iMessage
// and Outlook Safe Links fetch a URL to build a preview before any human sees
// it. If GET consumed the secret, most secrets would be destroyed in transit
// and the recipient's "already used" would be indistinguishable from a real
// interception.
//
// So: fetching the reveal page any number of times must leave the secret intact.
func TestGettingTheRevealPageNeverConsumesTheSecret(t *testing.T) {
h, _ := testApp(t)
ct := ciphertext("survives-the-previewers")
id := create(t, h, ct)
for i := range 5 {
code, _, _ := do(t, h, http.MethodGet, "/s/"+id, "")
if code != http.StatusOK {
t.Fatalf("GET /s/{id} attempt %d = %d, want 200", i, code)
}
}
code, body, raw := do(t, h, http.MethodPost, "/api/secrets/"+id+"/reveal", "")
if code != http.StatusOK {
t.Fatalf("reveal after 5 page loads = %d, want 200 — a GET consumed the secret: %s", code, raw)
}
if body["ciphertext"] != ct {
t.Fatal("the ciphertext changed across page loads")
}
}
// The reveal page must be identical for every id, including ids that were never
// minted. If it 404'd on an unknown id it would become an oracle for whether a
// link was ever real.
func TestTheRevealPageDoesNotDiscloseWhetherASecretExists(t *testing.T) {
h, _ := testApp(t)
id := create(t, h, ciphertext("real"))
_, _, real := do(t, h, http.MethodGet, "/s/"+id, "")
_, _, fake := do(t, h, http.MethodGet, "/s/"+strings.Repeat("A", 43), "")
_, _, junk := do(t, h, http.MethodGet, "/s/not-an-id", "")
if real != fake || real != junk {
t.Fatal("the reveal page differs between a real id, a well-formed unknown id, and junk — " +
"it must not disclose existence")
}
}
// A malformed id must produce the SAME 410 as a missing one. A 400 here would
// separate "not an id this service could mint" from "an id that is gone", which
// hands a probe a free classifier.
func TestMalformedAndMissingIDsAreIndistinguishable(t *testing.T) {
h, _ := testApp(t)
unknown, err := secret.NewID()
if err != nil {
t.Fatal(err)
}
var bodies []string
for _, id := range []string{
unknown.Value(), // well-formed, never stored
strings.Repeat("A", 43), // well-formed, never minted
"short", // wrong length
"!!!", // not base64url
} {
code, body, raw := do(t, h, http.MethodPost, "/api/secrets/"+id+"/reveal", "")
if code != http.StatusGone {
t.Fatalf("reveal(%q) = %d, want 410: %s", id, code, raw)
}
errObj, _ := body["error"].(map[string]any)
if errObj["code"] != "gone" {
t.Fatalf("reveal(%q) code = %v, want \"gone\"", id, errObj["code"])
}
bodies = append(bodies, errObj["message"].(string))
}
for i := range bodies {
if bodies[i] != bodies[0] {
t.Fatalf("the gone message differs between causes (%q vs %q) — they must be identical",
bodies[0], bodies[i])
}
}
}
func TestCreateRejectsWhatItCannotStore(t *testing.T) {
h, _ := testApp(t)
for _, tc := range []struct {
name string
body string
code string
}{
{"empty ciphertext", `{"ciphertext":"","ttl_seconds":3600}`, reasonEmpty},
{"not base64url", `{"ciphertext":"!!!!","ttl_seconds":3600}`, reasonInvalid},
{"oversized", `{"ciphertext":"` + strings.Repeat("A", secret.MaxCiphertextBytes+1) + `","ttl_seconds":3600}`, reasonTooLarge},
{"ttl too short", `{"ciphertext":"` + ciphertext("x") + `","ttl_seconds":60}`, reasonTTL},
{"ttl too long", `{"ciphertext":"` + ciphertext("x") + `","ttl_seconds":2592000}`, reasonTTL},
{"negative ttl", `{"ciphertext":"` + ciphertext("x") + `","ttl_seconds":-1}`, reasonTTL},
} {
t.Run(tc.name, func(t *testing.T) {
code, body, raw := do(t, h, http.MethodPost, "/api/secrets", tc.body)
if code != http.StatusUnprocessableEntity {
t.Fatalf("create = %d, want 422: %s", code, raw)
}
errObj, _ := body["error"].(map[string]any)
if errObj["code"] != tc.code {
t.Fatalf("error code = %v, want %q", errObj["code"], tc.code)
}
})
}
}
func TestOmittedTTLGetsTheDefault(t *testing.T) {
h, _ := testApp(t)
code, body, raw := do(t, h, http.MethodPost, "/api/secrets",
`{"ciphertext":"`+ciphertext("x")+`"}`)
if code != http.StatusCreated {
t.Fatalf("create without ttl_seconds = %d, want 201: %s", code, raw)
}
if got := body["ttl_seconds"].(float64); int(got) != int(secret.DefaultTTL.Seconds()) {
t.Fatalf("ttl_seconds = %v, want the default %v", got, secret.DefaultTTL.Seconds())
}
}
// There must be no way to hand hush a plaintext secret. If a field like
// "secret" or "plaintext" were ever accepted, the server would become able to
// read secrets and the guarantee in the README would quietly become a promise
// about our conduct instead of a property of the design.
func TestThereIsNoPlaintextIntakeField(t *testing.T) {
h, _ := testApp(t)
for _, body := range []string{
`{"secret":"hunter2","ttl_seconds":3600}`,
`{"plaintext":"hunter2","ttl_seconds":3600}`,
`{"ciphertext":"` + ciphertext("x") + `","plaintext":"hunter2"}`,
} {
code, _, raw := do(t, h, http.MethodPost, "/api/secrets", body)
if code == http.StatusCreated {
t.Fatalf("create accepted a plaintext field: %s -> %s", body, raw)
}
}
}
// The client-side crypto IS the product. If either page stops shipping it, the
// service silently becomes a plaintext store or a broken reader, and every
// other test here would still pass.
func TestPagesShipTheClientSideCrypto(t *testing.T) {
h, _ := testApp(t)
_, _, create := do(t, h, http.MethodGet, "/", "")
for _, want := range []string{
"AES-GCM", // the cipher
"crypto.subtle", // in the browser, not on the server
"generateKey", // the key is minted client-side
`"#" + sealed.key`, // and leaves only in the fragment
"/api/secrets", // ciphertext is what gets posted
} {
if !strings.Contains(create, want) {
t.Fatalf("the create page no longer contains %q — check it still encrypts client-side", want)
}
}
// The create page must never post a plaintext field.
for _, forbidden := range []string{`"plaintext"`, `"secret":`} {
if strings.Contains(create, forbidden) {
t.Fatalf("the create page mentions %s, which suggests it sends plaintext", forbidden)
}
}
_, _, reveal := do(t, h, http.MethodGet, "/s/"+strings.Repeat("A", 43), "")
for _, want := range []string{
"location.hash", // the key comes from the fragment
"crypto.subtle", // decryption is client-side
"/reveal", // and only POST consumes
"history.replaceState", // the key is dropped from the address bar after
} {
if !strings.Contains(reveal, want) {
t.Fatalf("the reveal page no longer contains %q", want)
}
}
}
func TestPagesAreNotCacheable(t *testing.T) {
h, _ := testApp(t)
for _, path := range []string{"/", "/s/" + strings.Repeat("A", 43)} {
r := httptest.NewRequest(http.MethodGet, path, nil)
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
// A cached reveal page in a shared proxy is a copy of a one-time URL.
if cc := w.Header().Get("Cache-Control"); !strings.Contains(cc, "no-store") {
t.Fatalf("%s Cache-Control = %q, want no-store", path, cc)
}
if ref := w.Header().Get("Referrer-Policy"); ref != "no-referrer" {
t.Fatalf("%s Referrer-Policy = %q, want no-referrer — a click could otherwise leak the URL", path, ref)
}
}
}

166
cmd/hushd/main.go Normal file
View File

@ -0,0 +1,166 @@
// Command hushd serves hush: one-time secret links.
//
// It stores ciphertext it cannot read. The encryption key is generated in the
// browser and travels only in the URL fragment, which browsers never transmit.
// See docs/ARCHITECTURE.md.
package main
import (
"context"
"errors"
"fmt"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/orchard9/go-chassis/chassis"
"github.com/orchard9/go-chassis/logging"
"github.com/orchard9/hush/internal/secret"
"github.com/orchard9/hush/internal/store"
"github.com/orchard9/hush/internal/web"
)
// service is the log corpus's `service` value and the metrics prefix. It is a
// member of a closed enum shared with every other orchard9 emitter — the
// cluster's Vector sink indexes `service` as a stream field, so a new value is
// a coordinated decision, not a free string.
const service = "hush"
func main() {
if err := run(); err != nil {
// Boot failures go to stderr as well as the log: if logging itself is
// what failed, the process must still say why before exiting.
fmt.Fprintf(os.Stderr, "hushd: %v\n", err)
os.Exit(1)
}
}
func run() error {
cfg, cfgErr := loadConfig()
// The logger is built before the config error is returned, so a
// misconfiguration is REPORTED through the same corpus as everything else
// rather than dying silently. cfg.Env is the zero value on failure, which
// logging.Env normalises.
log := logging.New(logging.Config{Service: service, Env: logging.Env(cfg.Env)})
logging.SetFallback(log)
if cfgErr != nil {
// critical + category boot is the fail-closed contract: this is a
// refusal to serve, which is the only thing critical is for.
logging.Critical(context.Background(), log, "boot.config_invalid",
"category", "boot", "error_type", "config_invalid", "error_msg", cfgErr.Error())
return cfgErr
}
pages, err := web.New()
if err != nil {
logging.Critical(context.Background(), log, "boot.templates_invalid",
"category", "boot", "error_type", "template_invalid", "error_msg", err.Error())
return err
}
rdb, err := store.NewRedis(cfg.RedisURL)
if err != nil {
// The URL is malformed — a config defect, not an outage. Refuse to boot
// rather than serve a store that can never work.
logging.Critical(context.Background(), log, "boot.store_invalid",
"category", "boot", "error_type", "store_invalid", "error_msg", err.Error())
return err
}
defer func() { _ = rdb.Close() }()
metrics := newMetrics()
metrics.Prime()
srv := &Server{cfg: cfg, store: rdb, pages: pages, metrics: metrics}
app := chassis.New(chassis.Config{
Service: service,
Env: cfg.Env,
Addr: fmt.Sprintf(":%d", cfg.Port),
// 128 KiB caps the body at the edge so an oversized upload is refused
// before a handler allocates it. The 64 KiB ciphertext cap is enforced
// separately in the handler; this is the outer bound including JSON
// framing.
MaxBodyBytes: 128 * 1024,
AllowOrigins: cfg.AllowOrigins,
Collectors: metrics.Collectors(),
}, log)
// Readiness doubles as the store_up gauge, so the alert on Redis
// reachability reads the same probe Kubernetes uses to route traffic —
// rather than a second, separately-drifting health notion.
app.Health("redis", func(ctx context.Context) error {
err := rdb.Ping(ctx)
if err != nil {
metrics.StoreUp.Set(0)
return err
}
metrics.StoreUp.Set(1)
return nil
})
// Pages: no storage access, no rate limit. A link previewer hitting either
// of these must be free and harmless.
app.Get("/", srv.handleCreatePage)
app.Get("/s/{id}", srv.handleRevealPage)
limiter := &redisLimiter{store: rdb, cfg: cfg, metrics: metrics}
app.Route("/api", func(r *chassis.Router) {
// Rate limit applies to create only. A reveal succeeds at most once per
// secret by construction, so there is nothing to throttle, and
// throttling would let one busy NAT block a colleague's delivery.
createMW := []chassis.Middleware{chassis.RateLimit(limiter, chassis.RateKey(cfg.TrustedProxyHops))}
if cfg.RequireAuthToCreate {
// The escape hatch for abuse. Reveal stays anonymous either way:
// the recipient is external and holds no credential.
createMW = append(createMW, chassis.RequireAuth(chassis.NewStaticToken(cfg.CreateToken)))
}
r.Post("/secrets", srv.handleCreate, createMW...)
r.Post("/secrets/{id}/reveal", srv.handleReveal)
})
log.Info("boot.ready", "category", "boot",
"port", cfg.Port,
"rate_limit_creates", cfg.RateLimitCreates,
"rate_limit_window_seconds", int64(cfg.RateLimitWindow.Seconds()),
"max_ciphertext_bytes", secret.MaxCiphertextBytes,
"default_ttl_seconds", int64(secret.DefaultTTL.Seconds()),
"require_auth_to_create", cfg.RequireAuthToCreate)
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := app.Run(ctx); err != nil && !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
}
// redisLimiter adapts the Redis fixed-window counter to chassis.RateLimiter and
// counts refusals, which is the abuse signal the alert rules watch.
type redisLimiter struct {
store *store.Redis
cfg Config
metrics *Metrics
}
func (l *redisLimiter) Allow(ctx context.Context, key string) (bool, time.Duration, error) {
ok, retry, err := l.store.AllowN(ctx, key, l.cfg.RateLimitCreates, l.cfg.RateLimitWindow)
if err != nil {
// Redis is unreachable. FAIL OPEN on the limiter specifically: the
// alternative is that a Redis blip turns the rate limiter into a total
// outage of a service whose whole job is delivering credentials during
// incidents. The store call immediately after will fail anyway if Redis
// is really down, so this cannot silently accept a secret it cannot
// store — it just refuses at the right layer, with the right error.
return true, 0, nil
}
if !ok {
l.metrics.Limited.Inc()
}
return ok, retry, nil
}

84
cmd/hushd/metrics.go Normal file
View File

@ -0,0 +1,84 @@
package main
import "github.com/prometheus/client_golang/prometheus"
// Metrics are hush's own series, registered on the chassis's private registry
// via chassis.Config.Collectors so /metrics is one scrape.
//
// Cardinality rule applied throughout: NO label ever carries a secret id, a
// client IP, or a URL path. A metric label lands in the time series index and
// stays there, so a per-secret label would be both a capability leak and an
// unbounded index. Every label below is a closed enum.
type Metrics struct {
Created prometheus.Counter
Revealed *prometheus.CounterVec // result: ok | gone
Rejected *prometheus.CounterVec // reason: closed enum, see rejection reasons
Bytes prometheus.Histogram
StoreUp prometheus.Gauge
Limited prometheus.Counter
}
// Rejection reasons. A closed set, mirrored by the API error codes, so
// `hush_secrets_rejected_total{reason="ciphertext_too_large"}` and the 422 a
// caller saw are the same vocabulary.
const (
reasonTooLarge = "ciphertext_too_large"
reasonInvalid = "ciphertext_invalid"
reasonEmpty = "ciphertext_empty"
reasonTTL = "ttl_out_of_range"
reasonMalformed = "malformed_request"
)
func newMetrics() *Metrics {
return &Metrics{
Created: prometheus.NewCounter(prometheus.CounterOpts{
Name: "hush_secrets_created_total",
Help: "Secrets accepted and stored.",
}),
Revealed: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "hush_secrets_revealed_total",
Help: "Reveal attempts by outcome. `gone` covers already-revealed, expired, never-existed and evicted, which the service does not distinguish.",
}, []string{"result"}),
Rejected: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "hush_secrets_rejected_total",
Help: "Create attempts refused by validation, by reason.",
}, []string{"reason"}),
Bytes: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "hush_secret_ciphertext_bytes",
Help: "Size of stored ciphertext.",
// Hand-picked rather than DefBuckets (which tops out at 10): this
// measures bytes up to a 64 KiB cap, so the buckets straddle
// credential-sized (hundreds of bytes) through the limit.
Buckets: []float64{256, 1024, 4096, 16384, 65536},
}),
StoreUp: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "hush_store_up",
Help: "1 when the last readiness probe reached Redis, 0 otherwise.",
}),
Limited: prometheus.NewCounter(prometheus.CounterOpts{
Name: "hush_rate_limited_total",
Help: "Requests refused by the create rate limit. A sustained rise is the abuse signal.",
}),
}
}
// Collectors is what gets handed to chassis.Config.Collectors. The chassis owns
// a private registry and exposes no accessor, so this slice is the only way
// hush's series reach /metrics.
func (m *Metrics) Collectors() []prometheus.Collector {
return []prometheus.Collector{m.Created, m.Revealed, m.Rejected, m.Bytes, m.StoreUp, m.Limited}
}
// Prime initialises the label combinations that alerting queries reference.
//
// Without this, `rate(hush_secrets_revealed_total{result="gone"}[15m])` returns
// no data until the first `gone` ever happens, and an alert written against a
// missing series is an alert that cannot fire. Priming makes the series exist
// at zero from boot.
func (m *Metrics) Prime() {
m.Revealed.WithLabelValues("ok")
m.Revealed.WithLabelValues("gone")
for _, r := range []string{reasonTooLarge, reasonInvalid, reasonEmpty, reasonTTL, reasonMalformed} {
m.Rejected.WithLabelValues(r)
}
}

249
deployments/k8s/hush.yaml Normal file
View File

@ -0,0 +1,249 @@
# hush on the orchard9 k3s cluster. Apply this BEFORE the first push, because
# the pipeline's deploy step runs `kubectl set image` and needs a Deployment to
# set it on.
#
# KUBECONFIG=~/.kube/orchard9-k3sf.yaml kubectl apply -f deployments/k8s/hush.yaml
#
# Everything hush needs is here: the credential, the workload, the Service, the
# network boundary and the public route.
---
# The Redis credential. hush connects as its OWN Redis ACL user, scoped to
# `~hush:*` with a minimal command set (+ping +set +getdel +incr +pexpire
# +select), so a bug in hush cannot read or write another tenant's keys and a
# compromise of hush cannot enumerate the keyspace.
#
# `+getdel` is the one that needs saying out loud: no other service's ACL user
# has it, because no other service needs an atomic read-and-destroy. Omitting it
# makes every reveal fail NOPERM while creates keep working — a service that
# accepts secrets and cannot deliver them.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hush-credentials
namespace: projects
spec:
refreshInterval: 1h
secretStoreRef:
name: gcp-secret-manager
kind: ClusterSecretStore
target:
name: hush-credentials
creationPolicy: Owner
dataFrom:
- extract:
key: k3sf-hush-credentials
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hush
namespace: projects
labels:
app: hush
spec:
# One replica is sufficient and not a bottleneck: every request is a single
# Redis round trip and all state lives in Redis, so this scales horizontally
# whenever it needs to. The rate limiter is already Redis-backed for exactly
# that reason.
replicas: 1
selector:
matchLabels:
app: hush
strategy:
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
template:
metadata:
labels:
app: hush
annotations:
# Metrics scraping. All three are required, and the port must EQUAL a
# declared containerPort as a string — vmagent's relabel config uses
# `keepequal`, so a mismatch silently drops the target with no error
# and no up=0 to alert on.
prometheus.io/scrape: "true"
prometheus.io/port: "18500"
prometheus.io/path: /metrics
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
# The chassis drains in two phases — flip readiness to 503, wait 5s for
# the load balancer to notice, then shut down with a 25s deadline. That is
# 30s, so the grace period must exceed it or the kubelet SIGKILLs mid-drain
# and in-flight reveals are lost.
terminationGracePeriodSeconds: 45
containers:
- name: hushd
# `bootstrap` is a placeholder, replaced by CI's `kubectl set image`
# with the commit-SHA tag on the first push. It is deliberately NOT
# `:latest`: the cluster's stable-controller-images admission policy
# refuses latest/main/master/dev/edge/canary/nightly/snapshot, because
# a floating tag cannot pin a rollback. Until the first pipeline run
# this image does not exist and the pod sits in ImagePullBackOff,
# which is the expected bootstrap state.
image: registry.threesix.ai/hush/api:bootstrap
ports:
- name: http
containerPort: 18500
env:
- name: APP_ENV
value: prod
- name: HUSH_PORT
value: "18500"
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: hush-credentials
key: REDIS_URL
# One Traefik hop sits in front, so the client IP is the last entry
# of X-Forwarded-For. Too high lets a caller spoof their IP past the
# rate limit; too low rate-limits the ingress itself and throttles
# every user together.
- name: HUSH_TRUSTED_PROXY_HOPS
value: "1"
- name: HUSH_RATE_LIMIT_CREATES
value: "30"
- name: HUSH_RATE_LIMIT_WINDOW
value: 10m
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
# Liveness stays 200 during the drain by design, so this restarts the
# pod only when the process is genuinely wedged — never merely because
# it is shutting down or because Redis is down.
livenessProbe:
httpGet: { path: /healthz, port: http }
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
# Readiness pings Redis. A Redis outage takes hush out of the Service
# rather than leaving it to serve 500s from a pod the load balancer
# still trusts.
readinessProbe:
httpGet: { path: /readyz, port: http }
initialDelaySeconds: 2
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
resources:
requests: { memory: 32Mi, cpu: 20m }
# 128Mi is generous for a service whose largest allocation is a
# 64 KiB ciphertext, and leaves headroom for Go's GC pacing.
limits: { memory: 128Mi, cpu: 500m }
---
apiVersion: v1
kind: Service
metadata:
name: hush
namespace: projects
labels:
app: hush
spec:
selector:
app: hush
ports:
- name: http
port: 80
targetPort: http
---
# Default-deny both directions, then open exactly what hush needs. Written as
# one policy because the ingress and egress rules are a single statement about
# this pod: Traefik in, DNS and Redis out, nothing else.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hush
namespace: projects
spec:
podSelector:
matchLabels:
app: hush
policyTypes: [Ingress, Egress]
ingress:
# Public traffic, via Traefik only.
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- { protocol: TCP, port: 18500 }
# vmagent scrapes /metrics on the pod IP directly. Without this rule the
# target is discovered and every scrape is connection-refused, which shows
# up as up=0 and fires ScrapeTargetDown rather than as a policy error.
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: observability
ports:
- { protocol: TCP, port: 18500 }
egress:
# DNS. Redis is a headless Service, so its name resolves to a pod IP that
# changes when Redis restarts — this must be re-resolvable, not cached.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
# Redis, and nothing else. hush makes no other outbound connection: it does
# not fetch, does not call a provider, and does not ship its own logs
# (Vector reads stdout off the node filesystem).
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: databases
ports:
- { protocol: TCP, port: 6379 }
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hush
namespace: projects
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts: [hush.threesix.ai]
secretName: hush-tls
rules:
- host: hush.threesix.ai
http:
paths:
# The paths are enumerated deliberately, and `/` is Exact rather than
# Prefix. A Prefix `/` would route EVERYTHING, publishing /metrics,
# /healthz and /readyz to the internet. /metrics leaks how many
# secrets are created and when; the others are just noise. Enumerating
# instead means Traefik 404s them at the edge and there is no
# basic-auth middleware to maintain and get wrong.
- path: /
pathType: Exact
backend:
service:
name: hush
port: { name: http }
- path: /s/
pathType: Prefix
backend:
service:
name: hush
port: { name: http }
- path: /api/
pathType: Prefix
backend:
service:
name: hush
port: { name: http }

180
docs/ARCHITECTURE.md Normal file
View File

@ -0,0 +1,180 @@
# Architecture
One Go binary, one Redis key per secret, no database. The interesting parts are
all about *where the key sits* and *what destroys the ciphertext*.
## Components
```
browser ──► Traefik ──► hushd (projects ns, 1 replica) ──► Redis (databases ns, db 5)
├─ stdout JSON ──► Vector (DaemonSet) ──► VictoriaLogs
└─ /metrics ─────► vmagent ──► vmsingle ──► vmalert ──► Alertmanager
```
`hushd` holds no durable state. Redis holds every secret and nothing else.
## The zero-knowledge split
```
create: plaintext ──[AES-256-GCM in the browser]──► ciphertext ──► POST /api/secrets
key ──────────────────────────────────────► URL fragment, never sent
reveal: POST /api/secrets/{id}/reveal ──► ciphertext ──[decrypt in browser]──► plaintext
key read from location.hash
```
The fragment is the whole trick. Per RFC 3986 §3.5 the fragment is a client-side
construct: browsers do not put it in the request line, so it never reaches
Traefik, hushd, Redis, an access log, or a proxy. hushd receives a 256-bit AES-GCM
ciphertext with a prepended 96-bit nonce and has no key material at any point.
Consequences worth stating plainly:
- A Redis dump is worthless. A hushd core dump is worthless. Our own operators
cannot read a secret, and neither can anyone who compromises the service.
- **A URL in someone's browser history contains the key.** The fragment is not
transmitted, but it *is* stored locally. This is the residual exposure and it
is why TTLs are short.
- There is deliberately no server-side-encryption fallback mode. A second mode
where the server sees plaintext would mean nobody could tell, from a link,
which guarantee they had.
## Why GET never touches storage
`GET /s/{id}` renders a static page and makes zero calls to Redis. It does not
even check whether the id exists.
That is not laziness — it is the only way to be correct in the presence of link
previewers. Slack, Teams, WhatsApp, iMessage and Outlook Safe Links fetch URLs
before a human sees them. Any design that destroys on `GET` destroys most secrets
in transit. Bot user-agent detection is a losing arms race; removing the
side effect from `GET` is not.
A secondary benefit: because `GET` does not look the id up, the reveal page cannot
leak whether an id exists. Existence is only ever answered by a `POST`, and that
answer is identical for missing, revealed and expired.
## Storage and destruction
One key per secret:
```
key hush:s:<id> id = 256 bits from crypto/rand, base64url (43 chars)
value <ciphertext> opaque bytes, ≤ 64 KiB
write SET key val EX <ttl> NX
read GETDEL key
```
`GETDEL` (Redis 6.2+; the cluster runs 7.4.8) is atomic, which is the reason it
is used instead of `GET` followed by `DEL`. Two people opening the same link
simultaneously cannot both receive the plaintext — exactly one `GETDEL` returns
the value and the other returns nil. A `GET`+`DEL` pair has a window between the
two commands where both callers succeed, and for a one-time secret that window is
the entire product.
`NX` on write means an id collision never overwrites an existing secret. At 256
bits of entropy a collision will not happen; the flag costs nothing and turns a
theoretical silent overwrite into a visible error.
TTL is Redis-native, so expiry needs no sweeper, no cron and nothing to wedge.
### Eviction is an availability risk, not a confidentiality one
The shared Redis runs `maxmemory-policy allkeys-lru` with `maxmemory 256MiB`.
Under memory pressure Redis may evict a hush key **before** its TTL fires. That
means a secret can become unavailable early.
It cannot become *more* available: eviction only ever deletes. So the failure mode
is "your recipient has to ask you again", never "the secret outlived its TTL" and
never "someone read it twice". For a secret courier that is the correct direction
to fail, and it is why `410 gone` deliberately does not distinguish causes — the
user-visible contract is identical either way.
Operationally this is watched via `HushRedisUnreachable` and the Redis memory
alerts, not by trying to tell eviction and reveal apart. See
[OPERATIONS.md](OPERATIONS.md).
## Identifiers and what gets logged
The id **is** the capability. Anyone holding it can reveal the secret, so it is
treated like a bearer token:
- Never logged. Not at debug, not in an error, not in a panic.
- The log correlation handle is `sid = sha256(id)[:12]` — enough to follow one
secret's create → reveal → gone across a corpus, useless for revealing it.
- Never in a metric label (that would put it in the time series index forever).
`internal/secret.ID.LogHandle()` is the only way to get a loggable form, and the
`ID` type does not implement `String()` or `MarshalText()`, so it cannot be
accidentally interpolated into a log line or JSON body. That is enforced by
`internal/secret/id_test.go`.
The chassis logger additionally redacts any field *named* `secret`, `token`,
`password`, `api_key`, `authorization` and friends. Field names here avoid those
words entirely (`ciphertext`, `sid`, `ttl_seconds`) so nothing depends on that
backstop.
## Request path
```
GET / create page (static HTML+JS, no storage access)
GET /s/{id} reveal page (static HTML+JS, no storage access)
POST /api/secrets store ciphertext rate limited
POST /api/secrets/{id}/reveal GETDEL, destroy, return once rate limited
GET /healthz liveness — 200 while draining
GET /readyz readiness — Redis PING, 503 while draining
GET /metrics Prometheus
```
Built on `github.com/orchard9/go-chassis`, which supplies routing, request ids,
the panic recovery envelope, RED metrics, secure headers, the two-phase drain,
and `/healthz`, `/readyz`, `/metrics`. hush contributes handlers, a store, a
rate limiter and templates — not a framework.
The public Ingress routes `/` (exact), `/s/` and `/api/` only. `/metrics`,
`/healthz` and `/readyz` share the port but are unreachable from the internet;
vmagent scrapes the pod IP directly. This is why there is no metrics basic-auth
middleware to maintain.
## Abuse posture
Create is anonymous by design, which makes the service a free blob host and a
phishing kit borrowing a `threesix.ai` name. Mitigations, all cheap:
| Control | Value |
|---|---|
| Ciphertext cap | 64 KiB, enforced before Redis |
| Request body cap | 128 KiB, enforced by the chassis at the edge |
| TTL clamp | 5m … 7d, out-of-range is a 422, not a silent clamp |
| Rate limit | 30 creates / 10 min / IP, Redis fixed-window |
| Id entropy | 256 bits — enumeration is not a threat model |
| No listing route | there is no way to ask "what secrets exist" |
| Identical `gone` | missing, revealed and expired are one response |
If it is ever abused, `HUSH_REQUIRE_AUTH=true` puts create behind the chassis
authenticator while leaving reveal anonymous — the asymmetry the design assumes.
Reveal must stay anonymous: the recipient is external and has no credential.
## Failure modes
| Failure | Behaviour |
|---|---|
| Redis down | `/readyz` 503, pod leaves the Service, creates and reveals 503. No secret is lost that was already written. |
| Redis evicts a key early | That link returns `410 gone`. Sender must re-send. |
| hushd restarts | Nothing lost; all state is in Redis. |
| Two simultaneous reveals | Exactly one wins, atomically. |
| Body over 128 KiB | 413 at the edge, never reaches a handler. |
| Ciphertext over 64 KiB | 422 `ciphertext_too_large`. |
| Malformed base64 | 422 `ciphertext_invalid`. hushd validates the encoding but cannot validate the plaintext. |
| Clock skew | TTL is Redis-relative, so skew between hushd and the browser cannot extend a secret's life. |
## What is deliberately absent
Accounts. Passphrases on top of the link. File uploads. Multi-read links. An
audit UI. Email delivery. Each is a real request and each doubles the surface.
The one with a genuine argument is **notify-on-read**: it confirms delivery and,
if it fires before the recipient says they opened it, that is a compromise
signal. It needs an email path, `notify` already exists to provide one, and it is
the first thing to add if hush proves useful.

21
go.mod Normal file
View File

@ -0,0 +1,21 @@
module github.com/orchard9/hush
go 1.26.0
require (
github.com/orchard9/go-chassis v0.1.0
github.com/prometheus/client_golang v1.24.0
github.com/redis/go-redis/v9 v9.22.0
)
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.0 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
go.uber.org/atomic v1.11.0 // indirect
golang.org/x/sys v0.47.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
)

50
go.sum Normal file
View File

@ -0,0 +1,50 @@
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ=
github.com/klauspost/compress v1.19.0/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/orchard9/go-chassis v0.1.0 h1:epi8NWfYIqhS0GkHMRLVS0v9//brI4Ithd7RyaoYxA4=
github.com/orchard9/go-chassis v0.1.0/go.mod h1:lhBicqc77tUEogh1jScEWEzxPqnsw2fpbzid/1bKVcQ=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.24.0 h1:5XStIklKuAtJSNpdD3s8XJj/Yv78IQmE1kbNk87JrAI=
github.com/prometheus/client_golang v1.24.0/go.mod h1:QcsNdotprC2nS4BTM2ucbcqxd2CeXTEa9jW7zHO9iDE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.70.0 h1:bcpru3tWPVnxGnETLgOV5jbp/JRXgYEyv65CuBLAMMI=
github.com/prometheus/common v0.70.0/go.mod h1:S/SFasQmgGiYH6C81LKCtYa8QACgthGg5zxL2udV7SY=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0=
github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

131
internal/secret/id.go Normal file
View File

@ -0,0 +1,131 @@
// Package secret holds hush's domain types and policy: identifiers, size and
// lifetime limits. It imports nothing outside the standard library, so the
// rules live in one place and are testable without Redis or HTTP.
package secret
import (
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"log/slog"
)
// IDBytes is the entropy behind a secret id. 256 bits makes enumeration a
// non-threat: an attacker guessing ids is not a scenario this service defends
// against with rate limits, it is a scenario arithmetic forecloses.
const IDBytes = 32
// idEncoding is base64url without padding, so an id is URL-safe, 43 characters,
// and needs no escaping in a path or a fragment.
var idEncoding = base64.RawURLEncoding
// ErrMalformedID is returned for a value that cannot be an id this service
// minted. Callers turn it into the same 410 as a missing secret — telling a
// caller that their id was well-formed but absent confirms the id space.
var ErrMalformedID = errors.New("malformed secret id")
// ID is a secret's identifier, and it IS the capability: anyone holding it can
// reveal the secret exactly once. It is therefore treated like a bearer token.
//
// Every accidental path prints a REDACTED form. An earlier version tried to
// prevent leaks by implementing no String() at all, which was wrong: Go's fmt
// prints unexported struct fields anyway, so `fmt.Sprintf("%v", id)` emitted
// the live id. Forbidding the method did not remove the leak, it only removed
// the chance to control it.
//
// So instead the safe form is the DEFAULT and the raw value needs an explicit
// call:
//
// String() -> "ID(a1b2c3d4e5f6)" fmt %v, %s, string concatenation
// LogValue() -> "a1b2c3d4e5f6" every slog call site
// MarshalJSON -> refuses a response struct cannot leak one silently
// Value() -> the raw id the two places it must escape
//
// Enforced by id_test.go, which fails if any of those starts emitting the raw
// value.
type ID struct {
raw string
}
// NewID mints a fresh identifier from crypto/rand. It returns an error rather
// than panicking: a service that cannot get entropy must refuse to mint a
// secret, not mint a guessable one.
func NewID() (ID, error) {
b := make([]byte, IDBytes)
if _, err := rand.Read(b); err != nil {
return ID{}, err
}
return ID{raw: idEncoding.EncodeToString(b)}, nil
}
// ParseID validates an id from a URL path. It checks the encoding and the
// decoded LENGTH, so a short-but-valid base64 string cannot become an id and
// widen the space a scanner has to cover.
func ParseID(s string) (ID, error) {
if len(s) != idEncoding.EncodedLen(IDBytes) {
return ID{}, ErrMalformedID
}
b, err := idEncoding.DecodeString(s)
if err != nil || len(b) != IDBytes {
return ID{}, ErrMalformedID
}
return ID{raw: s}, nil
}
// Value returns the raw id. Every call site is a place where the capability
// escapes, so there are deliberately few: the storage key and the created
// response body.
func (id ID) Value() string { return id.raw }
// IsZero reports whether this is the zero ID, which no minting path produces.
func (id ID) IsZero() bool { return id.raw == "" }
// LogHandle is the only form of an id that may be logged or reported: the first
// 12 hex characters of its SHA-256. It is stable, so one secret's create,
// reveal and gone lines correlate across the log corpus, and it is one-way, so
// a log reader cannot reveal the secret it refers to.
//
// 48 bits of a hash is not a secret-strength value and is not treated as one —
// it is a correlation handle. The preimage is 256 bits of entropy, so recovering
// an id from a handle is not feasible even though the handle is short.
func (id ID) LogHandle() string {
if id.raw == "" {
return ""
}
sum := sha256.Sum256([]byte(id.raw))
return hex.EncodeToString(sum[:])[:12]
}
// String is the redacted form, so `%v`, `%s` and string concatenation are all
// safe by default. Reaching the raw id requires Value().
func (id ID) String() string {
if id.raw == "" {
return "ID(zero)"
}
return "ID(" + id.LogHandle() + ")"
}
// LogValue makes every slog call site safe without the caller thinking about
// it: `log.Info("secret.created", "id", id)` emits the handle, not the
// capability. This is why the handlers can pass ids around without a review
// checklist for each log line.
func (id ID) LogValue() slog.Value { return slog.StringValue(id.LogHandle()) }
// MarshalJSON REFUSES rather than emitting either form.
//
// Emitting the raw id would leak a capability into any response struct that
// happened to embed an ID. Emitting the redacted form would be worse: it would
// produce a response that looks like it carries an id and does not, and the bug
// would surface as a broken link rather than a failed request. The two places
// an id legitimately reaches a client both call Value() explicitly.
func (id ID) MarshalJSON() ([]byte, error) {
return nil, errors.New("secret.ID must not be serialised: call Value() at the one site that needs it")
}
// StorageKey is the Redis key holding this secret's ciphertext. The `hush:`
// prefix is what the Redis ACL user is scoped to (`~hush:*`), so a bug that
// built a key outside this prefix would be refused by the server rather than
// touching another tenant's keyspace.
func (id ID) StorageKey() string { return "hush:s:" + id.raw }

190
internal/secret/id_test.go Normal file
View File

@ -0,0 +1,190 @@
package secret
import (
"bytes"
"encoding/json"
"fmt"
"log/slog"
"strings"
"testing"
)
// The id IS the capability, so every path that could print one is checked here.
//
// The first version of this test forbade String() outright, on the theory that
// a type with no Stringer cannot be interpolated. That was wrong and this test
// caught it: Go's fmt prints unexported struct fields regardless, so `%v` was
// emitting live ids. The design now makes the REDACTED form the default on
// every accidental path and requires Value() for the raw one.
//
// Read this test first if you are wondering why ID has a String() that throws
// away information and a MarshalJSON that refuses.
func TestNoAccidentalPathEmitsTheRawID(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
raw := id.Value()
for _, tc := range []struct {
name string
got string
}{
{"fmt %v", fmt.Sprintf("%v", id)},
{"fmt %s", fmt.Sprintf("%s", id)},
{"fmt %+v", fmt.Sprintf("%+v", id)},
{"fmt %#v of a wrapper struct", fmt.Sprintf("%v", struct{ ID ID }{id})},
{"String()", id.String()},
{"LogValue()", id.LogValue().String()},
{"inside a slice", fmt.Sprintf("%v", []ID{id})},
{"inside a map", fmt.Sprintf("%v", map[string]ID{"k": id})},
} {
t.Run(tc.name, func(t *testing.T) {
if strings.Contains(tc.got, raw) {
t.Fatalf("%s emitted the live id (%q) — this value would appear in a log line "+
"and anyone reading it could reveal the secret", tc.name, tc.got)
}
if !strings.Contains(tc.got, id.LogHandle()) {
t.Fatalf("%s = %q, which carries neither the id nor its handle; a log line "+
"with no handle cannot be correlated", tc.name, tc.got)
}
})
}
}
// A real slog handler, because that is where ids actually pass through.
func TestSlogEmitsTheHandleNotTheID(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
log := slog.New(slog.NewJSONHandler(&buf, nil))
// Every shape a call site might use, including the careless one.
log.Info("secret.created", "id", id)
log.Info("secret.created", "sid", id.LogHandle())
log.With("id", id).Info("secret.revealed")
log.Info("secret.gone", "ids", []ID{id})
out := buf.String()
if strings.Contains(out, id.Value()) {
t.Fatalf("slog output contains the live id:\n%s", out)
}
if !strings.Contains(out, id.LogHandle()) {
t.Fatalf("slog output contains no correlation handle:\n%s", out)
}
}
// Serialising an ID must FAIL rather than quietly emit either form. The raw
// value would leak a capability out of any response struct embedding an ID;
// the redacted value would produce a response that looks like it carries an id
// and does not, turning a server bug into a broken link for the recipient.
func TestMarshallingAnIDIsAnError(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
if b, err := json.Marshal(id); err == nil {
t.Fatalf("json.Marshal(ID) succeeded with %s; it must refuse", b)
}
// And from inside a struct, which is how it would actually happen.
if b, err := json.Marshal(struct {
ID ID `json:"id"`
}{id}); err == nil {
t.Fatalf("marshalling a struct containing an ID succeeded with %s; it must refuse", b)
}
}
func TestNewIDIsUnguessablyWideAndUnique(t *testing.T) {
seen := make(map[string]bool, 1000)
for range 1000 {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
if len(id.Value()) != 43 {
t.Fatalf("id %q is %d chars, want 43 (256 bits base64url unpadded)", id.Value(), len(id.Value()))
}
if seen[id.Value()] {
t.Fatal("NewID repeated a value in 1000 draws — entropy is broken")
}
seen[id.Value()] = true
}
}
func TestParseIDRoundTripsAndRefusesEverythingElse(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
back, err := ParseID(id.Value())
if err != nil {
t.Fatalf("ParseID rejected a freshly minted id: %v", err)
}
if back.Value() != id.Value() {
t.Fatalf("round trip changed the id: %q -> %q", id.Value(), back.Value())
}
for _, bad := range []struct{ name, in string }{
{"empty", ""},
{"too short", "abc"},
{"one char short", id.Value()[:42]},
{"one char long", id.Value() + "a"},
{"not base64url", strings.Repeat("!", 43)},
// A shorter secret encoded in a 43-char field would widen the space a
// scanner must cover, so the DECODED length is checked too.
{"padded base64", strings.Repeat("A", 40) + "==="},
{"path traversal", "../../etc/passwd"},
{"redis glob", strings.Repeat("*", 43)},
} {
t.Run(bad.name, func(t *testing.T) {
if _, err := ParseID(bad.in); err == nil {
t.Fatalf("ParseID(%q) accepted a value NewID cannot produce", bad.in)
}
})
}
}
func TestLogHandleIsStableShortAndNotTheID(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
h := id.LogHandle()
if len(h) != 12 {
t.Fatalf("LogHandle() = %q, want 12 hex chars", h)
}
if id.LogHandle() != h {
t.Fatal("LogHandle is not stable across calls, so log lines for one secret will not correlate")
}
if strings.Contains(h, id.Value()) || strings.Contains(id.Value(), h) {
t.Fatalf("LogHandle %q overlaps the raw id %q — it must be a hash, not a prefix", h, id.Value())
}
// Different ids must not collide into one handle, or correlation is wrong.
other, err := NewID()
if err != nil {
t.Fatal(err)
}
if other.LogHandle() == h {
t.Fatal("two ids produced the same LogHandle")
}
if (ID{}).LogHandle() != "" {
t.Fatal("the zero ID must produce an empty handle, not a hash of the empty string")
}
}
func TestStorageKeyStaysInsideTheACLPrefix(t *testing.T) {
id, err := NewID()
if err != nil {
t.Fatal(err)
}
// The Redis ACL user is scoped to ~hush:*. A key built outside that prefix
// is refused by the server, so this test is what keeps a rename from
// turning every write into a NOPERM at runtime.
if !strings.HasPrefix(id.StorageKey(), "hush:") {
t.Fatalf("StorageKey() = %q, which is outside the ~hush:* ACL scope", id.StorageKey())
}
}

80
internal/secret/policy.go Normal file
View File

@ -0,0 +1,80 @@
package secret
import (
"encoding/base64"
"errors"
"fmt"
"time"
)
// Size and lifetime policy. Every bound here is a product decision with a
// reason, not a tuning knob:
//
// - MaxCiphertextBytes keeps hush a courier for credentials rather than a file
// host. 64 KiB of AES-GCM holds roughly 48 KiB of plaintext, which is a very
// large credential and a very small file.
// - MinTTL exists because a link that expires before the recipient reads their
// messages is a support ticket, not a security win.
// - MaxTTL bounds exposure. The residual risk in this design is the key sitting
// in a browser history entry, and a week is as long as that is defensible.
const (
MaxCiphertextBytes = 64 * 1024
MinTTL = 5 * time.Minute
MaxTTL = 7 * 24 * time.Hour
DefaultTTL = 24 * time.Hour
)
// Validation failures. Each maps to one API error code, so a caller can branch
// on the cause without parsing prose.
var (
ErrCiphertextEmpty = errors.New("ciphertext is empty")
ErrCiphertextTooLarge = errors.New("ciphertext exceeds the size limit")
ErrCiphertextInvalid = errors.New("ciphertext is not valid base64url")
ErrTTLOutOfRange = errors.New("ttl is outside the permitted range")
)
// ciphertextEncoding matches what the browser produces: base64url, unpadded.
var ciphertextEncoding = base64.RawURLEncoding
// ValidateCiphertext checks what the server is ABLE to check. hush cannot
// verify that the bytes decrypt, because it has no key — by design. So it
// verifies the two things it can: that the encoding is what this service's
// clients produce, and that the size is inside the cap.
//
// The encoding check is not cosmetic. Without it, hush becomes a store for
// arbitrary bytes addressable by URL, which is a different and much less
// defensible service than the one described in the README.
func ValidateCiphertext(ciphertext string) error {
switch {
case ciphertext == "":
return ErrCiphertextEmpty
case len(ciphertext) > MaxCiphertextBytes:
// Measured on the encoded form, which is what is stored and what
// bounds memory. Checked BEFORE decoding so an oversized body is
// rejected without allocating its decoded copy.
return fmt.Errorf("%w: %d > %d bytes", ErrCiphertextTooLarge, len(ciphertext), MaxCiphertextBytes)
}
if _, err := ciphertextEncoding.DecodeString(ciphertext); err != nil {
return ErrCiphertextInvalid
}
return nil
}
// ResolveTTL turns a caller's requested lifetime into the one that will be
// used. Zero means "unspecified" and gets the default.
//
// An out-of-range value is an ERROR, never a silent clamp. A caller who asked
// for 30 days and got 7 without being told would believe their link outlives
// its actual expiry, and would find out when the recipient could not open it.
func ResolveTTL(requested time.Duration) (time.Duration, error) {
if requested == 0 {
return DefaultTTL, nil
}
if requested < MinTTL || requested > MaxTTL {
return 0, fmt.Errorf("%w: %s not in [%s, %s]", ErrTTLOutOfRange, requested, MinTTL, MaxTTL)
}
// Truncate to whole seconds: the wire format is seconds and Redis EX takes
// seconds, so keeping sub-second precision would make the expires_at we
// report disagree with the expiry Redis enforces.
return requested.Truncate(time.Second), nil
}

View File

@ -0,0 +1,91 @@
package secret
import (
"encoding/base64"
"errors"
"strings"
"testing"
"time"
)
func TestValidateCiphertext(t *testing.T) {
valid := base64.RawURLEncoding.EncodeToString([]byte("nonce+ciphertext bytes"))
for _, tc := range []struct {
name string
in string
want error
}{
{"a real base64url blob", valid, nil},
{"empty", "", ErrCiphertextEmpty},
{"not base64url", "!!!not base64!!!", ErrCiphertextInvalid},
// Standard base64 uses + and /, which are not URL-safe. Accepting them
// would mean two spellings of one ciphertext and a client that works in
// one browser and not another.
{"standard base64 alphabet", "YWJj+/8=", ErrCiphertextInvalid},
{"padded", "YWJjZA==", ErrCiphertextInvalid},
{"at the cap", strings.Repeat("A", MaxCiphertextBytes), nil},
{"one byte over the cap", strings.Repeat("A", MaxCiphertextBytes+1), ErrCiphertextTooLarge},
} {
t.Run(tc.name, func(t *testing.T) {
err := ValidateCiphertext(tc.in)
if !errors.Is(err, tc.want) {
t.Fatalf("ValidateCiphertext() = %v, want %v", err, tc.want)
}
})
}
}
// The size check must happen BEFORE the decode, or a 64 KiB+ body costs a
// decoded copy before being refused — which is the cheap half of a memory DoS.
func TestOversizedCiphertextIsRefusedWithoutDecoding(t *testing.T) {
// Deliberately not valid base64. If the implementation decoded first, this
// would come back as ErrCiphertextInvalid instead of ErrCiphertextTooLarge.
huge := strings.Repeat("!", MaxCiphertextBytes+1)
if err := ValidateCiphertext(huge); !errors.Is(err, ErrCiphertextTooLarge) {
t.Fatalf("ValidateCiphertext(oversized invalid) = %v, want ErrCiphertextTooLarge — "+
"the size gate must precede the decode", err)
}
}
func TestResolveTTL(t *testing.T) {
for _, tc := range []struct {
name string
in time.Duration
want time.Duration
err error
}{
{"unspecified takes the default", 0, DefaultTTL, nil},
{"at the minimum", MinTTL, MinTTL, nil},
{"at the maximum", MaxTTL, MaxTTL, nil},
{"a normal day", 24 * time.Hour, 24 * time.Hour, nil},
{"below the minimum", MinTTL - time.Second, 0, ErrTTLOutOfRange},
{"above the maximum", MaxTTL + time.Second, 0, ErrTTLOutOfRange},
{"negative", -time.Hour, 0, ErrTTLOutOfRange},
{"absurd", 3650 * 24 * time.Hour, 0, ErrTTLOutOfRange},
} {
t.Run(tc.name, func(t *testing.T) {
got, err := ResolveTTL(tc.in)
if !errors.Is(err, tc.err) {
t.Fatalf("ResolveTTL(%s) error = %v, want %v", tc.in, err, tc.err)
}
if err == nil && got != tc.want {
t.Fatalf("ResolveTTL(%s) = %s, want %s", tc.in, got, tc.want)
}
})
}
}
// An out-of-range TTL must be an ERROR and never a silent clamp. A caller who
// asked for 30 days and was quietly given 7 would believe their link outlives
// its real expiry, and would discover otherwise when the recipient could not
// open it.
func TestOutOfRangeTTLIsRefusedNotClamped(t *testing.T) {
got, err := ResolveTTL(30 * 24 * time.Hour)
if err == nil {
t.Fatalf("ResolveTTL(30d) silently returned %s instead of refusing", got)
}
if got != 0 {
t.Fatalf("ResolveTTL returned %s alongside an error; callers must not see a usable value", got)
}
}

80
internal/store/memory.go Normal file
View File

@ -0,0 +1,80 @@
package store
import (
"context"
"sync"
"time"
"github.com/orchard9/hush/internal/secret"
)
// Memory is an in-process Store for tests and `make dev` without Redis.
//
// It is NOT a deployment option, and the deployment path cannot select it: the
// store is chosen by whether REDIS_URL is set, and REDIS_URL is Required in
// config, so a misconfigured pod fails to boot rather than silently serving
// secrets from a store that dies with the process. This type exists so the
// handler tests do not need a container.
type Memory struct {
mu sync.Mutex
items map[string]memItem
now func() time.Time
}
type memItem struct {
ciphertext string
expiresAt time.Time
}
// NewMemory returns an empty store using the real clock.
func NewMemory() *Memory {
return &Memory{items: map[string]memItem{}, now: time.Now}
}
// NewMemoryAt returns a store driven by a caller-supplied clock, so expiry is
// testable without sleeping.
func NewMemoryAt(now func() time.Time) *Memory {
return &Memory{items: map[string]memItem{}, now: now}
}
func (m *Memory) Put(_ context.Context, id secret.ID, ciphertext string, ttl time.Duration) error {
m.mu.Lock()
defer m.mu.Unlock()
k := id.StorageKey()
// Mirror Redis SET NX semantics, including that an EXPIRED key is treated
// as absent and may be overwritten. A memory store that rejected a write
// against a stale entry would pass tests Redis fails.
if it, ok := m.items[k]; ok && m.now().Before(it.expiresAt) {
return ErrIDCollision
}
m.items[k] = memItem{ciphertext: ciphertext, expiresAt: m.now().Add(ttl)}
return nil
}
// Take mirrors GETDEL: the read and the delete happen under one lock, so the
// atomicity the one-time guarantee depends on holds here too and the
// concurrency test is meaningful against both implementations.
func (m *Memory) Take(_ context.Context, id secret.ID) (string, error) {
m.mu.Lock()
defer m.mu.Unlock()
k := id.StorageKey()
it, ok := m.items[k]
if !ok {
return "", ErrGone
}
delete(m.items, k)
if !m.now().Before(it.expiresAt) {
return "", ErrGone
}
return it.ciphertext, nil
}
func (m *Memory) Ping(context.Context) error { return nil }
func (m *Memory) Close() error { return nil }
// AllowN is an always-allow limiter: rate limiting is an abuse control on the
// public deployment, and silently enforcing one in tests would make handler
// tests order-dependent and flaky.
func (m *Memory) AllowN(context.Context, string, int, time.Duration) (bool, time.Duration, error) {
return true, 0, nil
}

117
internal/store/redis.go Normal file
View File

@ -0,0 +1,117 @@
package store
import (
"context"
"errors"
"fmt"
"time"
"github.com/redis/go-redis/v9"
"github.com/orchard9/hush/internal/secret"
)
// Redis is the production store. It keeps one key per secret and lets Redis own
// expiry, so nothing in hush sweeps, scans, or holds a timer.
type Redis struct {
client *redis.Client
}
// NewRedis dials Redis from a URL of the form
// redis://user:password@host:6379/5 — the ACL user, the password and the db
// index all ride in the URL, matching every other orchard9 service.
func NewRedis(url string) (*Redis, error) {
opt, err := redis.ParseURL(url)
if err != nil {
return nil, fmt.Errorf("parse redis url: %w", err)
}
// A secret write must not hang a request behind a slow dependency: the
// chassis request timeout would fire and the caller would see a 500 with no
// idea whether the secret was stored. Short, explicit timeouts make the
// failure fast and unambiguous.
opt.DialTimeout = 3 * time.Second
opt.ReadTimeout = 2 * time.Second
opt.WriteTimeout = 2 * time.Second
opt.MaxRetries = 2
return &Redis{client: redis.NewClient(opt)}, nil
}
// Put stores ciphertext with SET ... EX ttl NX.
//
// NX is what makes an id collision an error instead of a silent overwrite of a
// live secret. It cannot fire by chance at 256 bits, which is exactly why a
// false return is worth surfacing: it means something is wrong with id
// generation, and the alternative is destroying a secret somebody is waiting on.
func (r *Redis) Put(ctx context.Context, id secret.ID, ciphertext string, ttl time.Duration) error {
ok, err := r.client.SetNX(ctx, id.StorageKey(), ciphertext, ttl).Result()
if err != nil {
return fmt.Errorf("redis set: %w", err)
}
if !ok {
return ErrIDCollision
}
return nil
}
// Take reads and destroys in ONE command.
//
// GETDEL (Redis 6.2+) is atomic, which is the entire one-time guarantee. A
// GET followed by a DEL has a window between the two round trips where two
// simultaneous readers both get the plaintext, and for this service that window
// is the product. Redis 7.4.8 runs in the cluster; the ACL user must carry
// `+getdel` or every reveal returns NOPERM.
//
// redis.Nil covers all four "not available" causes and collapses to ErrGone —
// see the doc comment on ErrGone for why they are not separated.
func (r *Redis) Take(ctx context.Context, id secret.ID) (string, error) {
v, err := r.client.GetDel(ctx, id.StorageKey()).Result()
switch {
case errors.Is(err, redis.Nil):
return "", ErrGone
case err != nil:
return "", fmt.Errorf("redis getdel: %w", err)
}
return v, nil
}
// Ping backs /readyz. It is the reason a Redis outage takes the pod out of the
// Service rather than serving 500s from a pod the load balancer still trusts.
func (r *Redis) Ping(ctx context.Context) error {
return r.client.Ping(ctx).Err()
}
// Close shuts the pool down.
func (r *Redis) Close() error { return r.client.Close() }
// AllowN implements a fixed-window rate limit in Redis: INCR the window key,
// set its expiry on first use, and refuse once the count passes the limit.
//
// Fixed window rather than a sliding one because it costs two commands, needs
// no Lua, and the failure it permits — up to 2x the limit across a window
// boundary — is irrelevant for an abuse control whose job is to stop bulk
// automation, not to meter precisely.
//
// It lives in Redis rather than in process memory so the limit still holds if
// hush is ever scaled past one replica, and so a restart cannot be used to
// reset it.
func (r *Redis) AllowN(ctx context.Context, key string, limit int, window time.Duration) (bool, time.Duration, error) {
full := "hush:rl:" + key
pipe := r.client.Pipeline()
incr := pipe.Incr(ctx, full)
// NX so a long-running window is not extended by later requests inside it;
// without it a steady stream of calls would push the expiry forward forever
// and the window would never reset.
pipe.ExpireNX(ctx, full, window)
if _, err := pipe.Exec(ctx); err != nil {
return false, 0, fmt.Errorf("redis ratelimit: %w", err)
}
count := incr.Val()
if count > int64(limit) {
retry, err := r.client.PTTL(ctx, full).Result()
if err != nil || retry < 0 {
retry = window
}
return false, retry, nil
}
return true, 0, nil
}

50
internal/store/store.go Normal file
View File

@ -0,0 +1,50 @@
// Package store persists ciphertext for at most a TTL and destroys it on the
// first successful read.
package store
import (
"context"
"errors"
"time"
"github.com/orchard9/hush/internal/secret"
)
// ErrGone is returned by Take for every reason a secret is not available: it
// never existed, it was already revealed, it expired, or Redis evicted it early
// under memory pressure.
//
// The causes are deliberately NOT distinguished. Not because they are hard to
// tell apart, but because telling them apart is the leak: a caller who can
// separate "never existed" from "already revealed" can confirm that a given
// link was real, which is information about someone else's secret. The API maps
// this one error to one 410 for all four cases.
var ErrGone = errors.New("secret is gone")
// ErrIDCollision means Put found the key already occupied. At 256 bits of id
// entropy this cannot happen by chance, so it means a bug or a repeated id, and
// it is surfaced rather than silently overwriting a live secret.
var ErrIDCollision = errors.New("secret id already exists")
// Store is the whole persistence contract. Two methods, both destructive-safe:
// there is no Get, no List, and no Exists — a store that could answer "does
// this id exist" without consuming it would let the reveal page leak existence,
// and a store that could list would make a compromise catastrophic instead of
// merely bad.
type Store interface {
// Put writes ciphertext under id, expiring after ttl. It must fail rather
// than overwrite an existing key.
Put(ctx context.Context, id secret.ID, ciphertext string, ttl time.Duration) error
// Take returns the ciphertext and destroys it ATOMICALLY. Two concurrent
// callers must not both receive a value; exactly one gets it and the other
// gets ErrGone. This atomicity is the one-time property — an implementation
// that reads then deletes has a window in which both callers succeed.
Take(ctx context.Context, id secret.ID) (string, error)
// Ping reports whether the backing store is usable, for readiness.
Ping(ctx context.Context) error
// Close releases resources.
Close() error
}

View File

@ -0,0 +1,246 @@
package store_test
import (
"context"
"errors"
"os"
"sync"
"testing"
"time"
"github.com/orchard9/hush/internal/secret"
"github.com/orchard9/hush/internal/store"
)
// One contract, two implementations. The suite runs against Memory always and
// against Redis whenever HUSH_TEST_REDIS_URL is set (`make test-redis`, and CI
// where a Redis service is available).
//
// Running the SAME assertions against both is the point: Memory exists so
// handler tests need no container, and it is only trustworthy if it is held to
// the behaviour Redis actually has — including that an expired key may be
// overwritten and that Take is atomic.
func eachStore(t *testing.T, fn func(t *testing.T, s store.Store)) {
t.Helper()
t.Run("memory", func(t *testing.T) {
fn(t, store.NewMemory())
})
url := os.Getenv("HUSH_TEST_REDIS_URL")
if url == "" {
t.Log("HUSH_TEST_REDIS_URL unset: skipping the Redis half of the contract suite")
return
}
t.Run("redis", func(t *testing.T) {
r, err := store.NewRedis(url)
if err != nil {
t.Fatalf("dial redis: %v", err)
}
t.Cleanup(func() { _ = r.Close() })
if err := r.Ping(context.Background()); err != nil {
t.Fatalf("ping redis at %s: %v", url, err)
}
fn(t, r)
})
}
func newID(t *testing.T) secret.ID {
t.Helper()
id, err := secret.NewID()
if err != nil {
t.Fatal(err)
}
return id
}
func TestPutThenTakeReturnsTheCiphertextExactlyOnce(t *testing.T) {
eachStore(t, func(t *testing.T, s store.Store) {
ctx := context.Background()
id := newID(t)
const ct = "bm9uY2UtYW5kLWNpcGhlcnRleHQ"
if err := s.Put(ctx, id, ct, time.Minute); err != nil {
t.Fatalf("Put: %v", err)
}
got, err := s.Take(ctx, id)
if err != nil {
t.Fatalf("first Take: %v", err)
}
if got != ct {
t.Fatalf("first Take = %q, want %q", got, ct)
}
// The whole product: the second read must find nothing.
if _, err := s.Take(ctx, id); !errors.Is(err, store.ErrGone) {
t.Fatalf("second Take = %v, want ErrGone — the secret was not destroyed", err)
}
})
}
func TestTakeOfAnUnknownIDIsGone(t *testing.T) {
eachStore(t, func(t *testing.T, s store.Store) {
if _, err := s.Take(context.Background(), newID(t)); !errors.Is(err, store.ErrGone) {
t.Fatalf("Take(unknown) = %v, want ErrGone", err)
}
})
}
func TestPutRefusesToOverwriteALiveSecret(t *testing.T) {
eachStore(t, func(t *testing.T, s store.Store) {
ctx := context.Background()
id := newID(t)
if err := s.Put(ctx, id, "first", time.Minute); err != nil {
t.Fatal(err)
}
// At 256 bits this cannot happen by chance, so if it ever does it is an
// id-generation bug. Destroying the live secret instead of reporting it
// would lose a secret someone is waiting on.
if err := s.Put(ctx, id, "second", time.Minute); !errors.Is(err, store.ErrIDCollision) {
t.Fatalf("second Put = %v, want ErrIDCollision", err)
}
got, err := s.Take(ctx, id)
if err != nil || got != "first" {
t.Fatalf("Take after refused overwrite = (%q, %v), want (\"first\", nil)", got, err)
}
})
}
// Exactly one winner under concurrency. This is why the store contract demands
// an atomic read-and-destroy: a GET followed by a DEL has a window in which two
// readers both receive the plaintext, and for a one-time secret that window is
// the entire guarantee.
func TestConcurrentTakesProduceExactlyOneWinner(t *testing.T) {
eachStore(t, func(t *testing.T, s store.Store) {
ctx := context.Background()
const racers = 32
for round := range 20 {
id := newID(t)
if err := s.Put(ctx, id, "only-once", time.Minute); err != nil {
t.Fatalf("round %d Put: %v", round, err)
}
var (
wg sync.WaitGroup
mu sync.Mutex
wins int
gone int
othererr error
start = make(chan struct{})
)
for range racers {
wg.Add(1)
go func() {
defer wg.Done()
<-start // release them together to maximise the overlap
v, err := s.Take(ctx, id)
mu.Lock()
defer mu.Unlock()
switch {
case err == nil && v == "only-once":
wins++
case errors.Is(err, store.ErrGone):
gone++
default:
othererr = err
}
}()
}
close(start)
wg.Wait()
if othererr != nil {
t.Fatalf("round %d: unexpected error from Take: %v", round, othererr)
}
if wins != 1 {
t.Fatalf("round %d: %d goroutines received the secret, want exactly 1 (%d saw gone)", round, wins, gone)
}
if gone != racers-1 {
t.Fatalf("round %d: %d saw gone, want %d", round, gone, racers-1)
}
}
})
}
func TestAnExpiredSecretIsGoneAndItsIDIsReusable(t *testing.T) {
// Driven by a fake clock so this does not sleep. The Redis half of the
// contract is covered by TestRedisHonoursTTL below, which uses a short real
// TTL because Redis owns that clock.
now := time.Now()
s := store.NewMemoryAt(func() time.Time { return now })
ctx := context.Background()
id := newID(t)
if err := s.Put(ctx, id, "vanishing", time.Minute); err != nil {
t.Fatal(err)
}
now = now.Add(time.Minute + time.Second)
if _, err := s.Take(ctx, id); !errors.Is(err, store.ErrGone) {
t.Fatalf("Take after expiry = %v, want ErrGone", err)
}
// Redis treats an expired key as absent, so Put must succeed here. A
// memory store that refused would pass tests Redis fails.
if err := s.Put(ctx, id, "reused", time.Minute); err != nil {
t.Fatalf("Put over an expired key = %v, want nil (Redis SET NX succeeds on an expired key)", err)
}
}
func TestRedisHonoursTTL(t *testing.T) {
url := os.Getenv("HUSH_TEST_REDIS_URL")
if url == "" {
t.Skip("HUSH_TEST_REDIS_URL unset")
}
r, err := store.NewRedis(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = r.Close() })
ctx := context.Background()
id := newID(t)
// Redis EX takes whole seconds, so 1s is the shortest observable TTL.
if err := r.Put(ctx, id, "brief", time.Second); err != nil {
t.Fatal(err)
}
time.Sleep(1500 * time.Millisecond)
if _, err := r.Take(ctx, id); !errors.Is(err, store.ErrGone) {
t.Fatalf("Take after the TTL elapsed = %v, want ErrGone", err)
}
}
func TestRedisRateLimitCountsWithinAWindowAndThenRefuses(t *testing.T) {
url := os.Getenv("HUSH_TEST_REDIS_URL")
if url == "" {
t.Skip("HUSH_TEST_REDIS_URL unset")
}
r, err := store.NewRedis(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = r.Close() })
ctx := context.Background()
// A unique key per run so a re-run is not throttled by the previous one.
id := newID(t)
key := "test-" + id.LogHandle()
for i := 1; i <= 3; i++ {
ok, _, err := r.AllowN(ctx, key, 3, 5*time.Second)
if err != nil || !ok {
t.Fatalf("call %d: AllowN = (%v, %v), want allowed", i, ok, err)
}
}
ok, retry, err := r.AllowN(ctx, key, 3, 5*time.Second)
if err != nil {
t.Fatal(err)
}
if ok {
t.Fatal("the 4th call in a limit-3 window was allowed")
}
if retry <= 0 || retry > 5*time.Second {
t.Fatalf("retryAfter = %s, want a positive value inside the window", retry)
}
}

View File

@ -0,0 +1,94 @@
{{define "base.html"}}<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>hush</title>
<!-- No external origins: no CDN, no font host, no analytics. A third-party
script on this page could read the key out of the fragment, so the CSP
forbids one existing. 'unsafe-inline' covers the inline script and style
below, which are same-document and reviewed in this repo. -->
<meta http-equiv="Content-Security-Policy"
content="default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; connect-src 'self'; form-action 'none'; base-uri 'none'; frame-ancestors 'none'">
<meta name="referrer" content="no-referrer">
<style>
:root{color-scheme:dark;--bg:#0b0d10;--fg:#e7ebf0;--dim:#8b95a3;--line:#1e242c;--accent:#7fd1ae;--warn:#f0b866;--bad:#e8737d}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.55 ui-monospace,SFMono-Regular,Menlo,monospace;display:flex;min-height:100vh;align-items:center;justify-content:center;padding:24px}
main{width:100%;max-width:620px}
h1{font-size:20px;margin:0 0 4px;letter-spacing:.02em}
h1 span{color:var(--accent)}
p.lede{color:var(--dim);margin:0 0 24px;font-size:13px}
label{display:block;font-size:12px;color:var(--dim);margin:16px 0 6px;text-transform:uppercase;letter-spacing:.08em}
textarea,input,select,button{font:inherit;color:var(--fg);background:#12161b;border:1px solid var(--line);border-radius:6px;padding:10px 12px;width:100%}
textarea{min-height:150px;resize:vertical}
button{background:var(--accent);color:#06231a;border:0;font-weight:600;cursor:pointer;margin-top:20px;padding:12px}
button:hover{filter:brightness(1.08)}
button:disabled{opacity:.5;cursor:not-allowed;filter:none}
button.secondary{background:#12161b;color:var(--fg);border:1px solid var(--line);font-weight:400}
.row{display:flex;gap:12px}
.row>*{flex:1}
.out{margin-top:24px;padding:14px;border:1px solid var(--line);border-radius:6px;background:#0f1418;word-break:break-all;font-size:13px}
.note{color:var(--dim);font-size:12px;margin-top:10px}
.err{color:var(--bad)}
.warn{color:var(--warn)}
.ok{color:var(--accent)}
.hide{display:none}
footer{margin-top:28px;color:var(--dim);font-size:11px;border-top:1px solid var(--line);padding-top:12px}
a{color:var(--dim)}
</style>
</head>
<body>
<main>
{{template "content" .}}
<footer>
Encrypted in your browser. The key travels in the link's <code>#fragment</code>,
which browsers never send to a server — hush stores ciphertext it cannot open.
The link works once.
</footer>
</main>
<script>
// Shared crypto. AES-256-GCM via WebCrypto; the key is generated here, never
// transmitted, and carried only in the URL fragment.
//
// b64u: base64url without padding, matching Go's base64.RawURLEncoding on the
// server so the wire format has exactly one spelling.
const b64u = {
enc(bytes) {
let s = ""; for (const b of bytes) s += String.fromCharCode(b);
return btoa(s).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
},
dec(str) {
const p = str.replace(/-/g, "+").replace(/_/g, "/");
const raw = atob(p + "=".repeat((4 - (p.length % 4)) % 4));
const out = new Uint8Array(raw.length);
for (let i = 0; i < raw.length; i++) out[i] = raw.charCodeAt(i);
return out;
},
};
async function seal(plaintext) {
const key = await crypto.subtle.generateKey({ name: "AES-GCM", length: 256 }, true, ["encrypt", "decrypt"]);
// 96-bit nonce is the AES-GCM standard size and is generated per secret. It
// is prepended to the ciphertext rather than sent separately so a stored blob
// is self-contained.
const nonce = crypto.getRandomValues(new Uint8Array(12));
const body = new Uint8Array(await crypto.subtle.encrypt(
{ name: "AES-GCM", iv: nonce }, key, new TextEncoder().encode(plaintext)));
const blob = new Uint8Array(nonce.length + body.length);
blob.set(nonce, 0); blob.set(body, nonce.length);
const rawKey = new Uint8Array(await crypto.subtle.exportKey("raw", key));
return { ciphertext: b64u.enc(blob), key: b64u.enc(rawKey) };
}
async function open(ciphertext, keyStr) {
const blob = b64u.dec(ciphertext);
const key = await crypto.subtle.importKey("raw", b64u.dec(keyStr), "AES-GCM", false, ["decrypt"]);
const plain = await crypto.subtle.decrypt(
{ name: "AES-GCM", iv: blob.slice(0, 12) }, key, blob.slice(12));
return new TextDecoder().decode(plain);
}
</script>
{{template "script" .}}
</body>
</html>{{end}}

View File

@ -0,0 +1,118 @@
{{define "content"}}
<h1>hush<span>.</span></h1>
<p class="lede">Paste a secret. Get a link that works once.</p>
<div id="form">
<label for="secret">Secret</label>
<textarea id="secret" autofocus autocomplete="off" spellcheck="false"
placeholder="API key, password, connection string…"></textarea>
<div class="row">
<div>
<label for="ttl">Expires after</label>
<select id="ttl">
<option value="900">15 minutes</option>
<option value="3600">1 hour</option>
<option value="86400" selected>24 hours</option>
<option value="259200">3 days</option>
<option value="604800">7 days</option>
</select>
</div>
</div>
<button id="go">Create link</button>
<p class="note" id="msg"></p>
</div>
<div id="result" class="hide">
<label>Your one-time link</label>
<div class="out" id="link"></div>
<div class="row">
<button id="copy">Copy link</button>
<button id="again" class="secondary">Create another</button>
</div>
<p class="note warn">
This is shown once and is not recoverable — hush cannot rebuild it, because
the key it contains was never sent to the server. Copy it now.
</p>
</div>
{{end}}
{{define "script"}}
<script>
const $ = (id) => document.getElementById(id);
const MAX = {{.MaxCiphertextBytes}};
async function create() {
const text = $("secret").value;
if (!text) { return fail("Nothing to send."); }
$("go").disabled = true;
$("msg").className = "note";
$("msg").textContent = "Encrypting…";
let sealed;
try {
sealed = await seal(text);
} catch (e) {
return fail("Encryption failed in this browser: " + e.message);
}
// Check the size of the CIPHERTEXT, which is what the server caps, so the
// message names the same number the server would reject on.
if (sealed.ciphertext.length > MAX) {
return fail("Too large: " + sealed.ciphertext.length + " > " + MAX + " bytes encrypted.");
}
let res, body;
try {
res = await fetch("/api/secrets", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ciphertext: sealed.ciphertext, ttl_seconds: Number($("ttl").value) }),
});
body = await res.json();
} catch (e) {
return fail("Could not reach hush: " + e.message);
}
if (!res.ok) {
return fail(body && body.error ? body.error.message : "Server refused the secret (" + res.status + ").");
}
// The key goes in the fragment and ONLY in the fragment. Assembling the URL
// here — not on the server — is what keeps the key client-side.
const url = location.origin + "/s/" + body.id + "#" + sealed.key;
$("link").textContent = url;
$("form").classList.add("hide");
$("result").classList.remove("hide");
$("copy").focus();
}
function fail(m) {
$("go").disabled = false;
$("msg").className = "note err";
$("msg").textContent = m;
}
$("go").addEventListener("click", create);
$("secret").addEventListener("keydown", (e) => {
if ((e.metaKey || e.ctrlKey) && e.key === "Enter") create();
});
$("copy").addEventListener("click", async () => {
try {
await navigator.clipboard.writeText($("link").textContent);
$("copy").textContent = "Copied";
setTimeout(() => ($("copy").textContent = "Copy link"), 1500);
} catch {
// Clipboard needs a permission this browser withheld; selecting the text
// is a working fallback rather than a dead button.
const r = document.createRange();
r.selectNodeContents($("link"));
getSelection().removeAllRanges();
getSelection().addRange(r);
$("copy").textContent = "Selected — press copy";
}
});
$("again").addEventListener("click", () => location.assign("/"));
</script>
{{end}}

View File

@ -0,0 +1,118 @@
{{define "content"}}
<h1>hush<span>.</span></h1>
<p class="lede">Someone sent you a secret. It can be opened once.</p>
<div id="gate">
<p class="note">
Nothing has been read yet. This page has not touched the secret — opening it
is the button below, so a link preview in a chat app cannot consume it.
</p>
<button id="go">Reveal the secret</button>
<p class="note" id="msg"></p>
</div>
<div id="result" class="hide">
<label>The secret</label>
<div class="out" id="plain"></div>
<div class="row">
<button id="copy">Copy</button>
</div>
<p class="note warn">
Destroyed. Reloading this page will not show it again — copy it now.
</p>
</div>
{{end}}
{{define "script"}}
<script>
const $ = (id) => document.getElementById(id);
// The id comes from the path and the key from the fragment. Neither is
// interpolated by the server, so this page renders identically for every secret
// and reflects nothing.
const id = location.pathname.replace(/^\/s\//, "");
const key = location.hash.slice(1);
if (!key) {
$("go").disabled = true;
$("msg").className = "note err";
$("msg").textContent =
"This link is missing its key — the part after '#'. Chat apps and email " +
"clients sometimes truncate it. Ask the sender for the full link; the " +
"secret is intact and has not been opened.";
}
async function reveal() {
$("go").disabled = true;
$("msg").className = "note";
$("msg").textContent = "Opening…";
let res, body;
try {
res = await fetch("/api/secrets/" + encodeURIComponent(id) + "/reveal", { method: "POST" });
body = await res.json();
} catch (e) {
// The secret is very likely consumed at this point, so do not offer a retry
// that would report "gone" and read as a lie about what happened.
$("msg").className = "note err";
$("msg").textContent = "Could not reach hush: " + e.message +
". If the request left your browser, the secret is already destroyed.";
return;
}
if (res.status === 410) {
$("msg").className = "note err";
$("msg").textContent =
"Gone. This link was already opened, expired, or never existed. " +
"If you have not opened it yourself, assume someone else did and ask the " +
"sender to rotate the secret.";
return;
}
if (!res.ok) {
$("go").disabled = false;
$("msg").className = "note err";
$("msg").textContent = body && body.error ? body.error.message : "Server error (" + res.status + ").";
return;
}
let text;
try {
text = await open(body.ciphertext, key);
} catch {
// Decryption failed AFTER the server destroyed the ciphertext, so there is
// nothing to retry. Say so, because the alternative is a user reloading
// forever against a secret that no longer exists.
$("msg").className = "note err";
$("msg").textContent =
"The key in this link does not open this secret, and the ciphertext has " +
"now been destroyed. The link was probably altered in transit. Ask the " +
"sender to create a new one.";
return;
}
$("plain").textContent = text;
$("gate").classList.add("hide");
$("result").classList.remove("hide");
// Drop the key from the address bar so a screenshot, a shoulder-surfer or a
// later copy of the URL does not carry it. The secret is already destroyed
// server-side, so this only reduces incidental exposure.
history.replaceState(null, "", location.pathname);
$("copy").focus();
}
$("go").addEventListener("click", reveal);
$("copy").addEventListener("click", async () => {
try {
await navigator.clipboard.writeText($("plain").textContent);
$("copy").textContent = "Copied";
setTimeout(() => ($("copy").textContent = "Copy"), 1500);
} catch {
const r = document.createRange();
r.selectNodeContents($("plain"));
getSelection().removeAllRanges();
getSelection().addRange(r);
$("copy").textContent = "Selected — press copy";
}
});
</script>
{{end}}

73
internal/web/web.go Normal file
View File

@ -0,0 +1,73 @@
// Package web serves hush's two pages. Both are static: they read no storage,
// so a link previewer fetching either one cannot destroy a secret.
package web
import (
"embed"
"fmt"
"html/template"
"net/http"
)
//go:embed templates/*.html
var files embed.FS
// Pages renders the create and reveal pages. Templates are embedded, so the
// container carries no template directory to go missing at runtime.
type Pages struct {
create *template.Template
reveal *template.Template
}
// Data is everything a page needs. The limits are passed through so the browser
// enforces the same caps the server does and a user learns their secret is too
// large before uploading it, not after.
type Data struct {
MaxCiphertextBytes int
DefaultTTLSeconds int
MinTTLSeconds int
MaxTTLSeconds int
}
// New parses the embedded templates. It fails at boot rather than on first
// request: a template error is a build defect and should not wait for traffic
// to surface.
func New() (*Pages, error) {
create, err := template.ParseFS(files, "templates/base.html", "templates/create.html")
if err != nil {
return nil, fmt.Errorf("parse create template: %w", err)
}
reveal, err := template.ParseFS(files, "templates/base.html", "templates/reveal.html")
if err != nil {
return nil, fmt.Errorf("parse reveal template: %w", err)
}
return &Pages{create: create, reveal: reveal}, nil
}
// Create writes the create page.
func (p *Pages) Create(w http.ResponseWriter, d Data) error {
return render(w, p.create, d)
}
// Reveal writes the reveal page.
//
// The secret id is NOT passed in and is NOT interpolated into the HTML. The
// page reads it from location.pathname in the browser, alongside the key it
// reads from location.hash. That keeps the template free of any value that
// could be reflected, and means this handler needs no escaping decisions about
// a capability.
func (p *Pages) Reveal(w http.ResponseWriter, d Data) error {
return render(w, p.reveal, d)
}
func render(w http.ResponseWriter, t *template.Template, d Data) error {
// no-store on both pages: a cached create page is harmless, but a cached
// reveal page in a shared proxy would be a copy of a one-time URL.
w.Header().Set("Cache-Control", "no-store, max-age=0")
w.Header().Set("Content-Type", "text/html; charset=utf-8")
// Referrer-Policy is load-bearing here, not boilerplate: without it a click
// on any link from the reveal page could send the full URL — including the
// fragment-adjacent path — to a third party.
w.Header().Set("Referrer-Policy", "no-referrer")
return t.ExecuteTemplate(w, "base.html", d)
}

94
scripts/smoke.sh Executable file
View File

@ -0,0 +1,94 @@
#!/usr/bin/env bash
# End-to-end proof against a RUNNING hushd, doing the real client-side crypto.
#
# This is not a mock: it generates an AES-256-GCM key, encrypts a plaintext,
# posts only the ciphertext, reveals it once, decrypts it, and compares. Then it
# checks the three properties the design rests on:
#
# 1. a second reveal is 410 gone
# 2. GET on the reveal page does NOT consume the secret (link previewers)
# 3. a malformed id and a missing id are indistinguishable
#
# Usage: BASE=http://127.0.0.1:18500 ./scripts/smoke.sh
set -euo pipefail
BASE="${BASE:-http://127.0.0.1:18500}"
PLAINTEXT="${PLAINTEXT:-hunter2-$(date +%s)-$RANDOM}"
pass() { printf ' \033[32mok\033[0m %s\n' "$1"; }
fail() { printf ' \033[31mFAIL\033[0m %s\n' "$1"; exit 1; }
command -v python3 >/dev/null || { echo "python3 is required for the crypto half"; exit 1; }
echo "hush smoke against $BASE"
# --- encrypt exactly as the browser does -----------------------------------
# AES-256-GCM, 96-bit nonce prepended, base64url unpadded. Same wire format as
# templates/base.html, which is what makes this a real client.
read -r CIPHERTEXT KEY <<<"$(python3 - "$PLAINTEXT" <<'PY'
import base64, os, sys
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
key = AESGCM.generate_key(bit_length=256)
nonce = os.urandom(12)
blob = nonce + AESGCM(key).encrypt(nonce, sys.argv[1].encode(), None)
b = lambda x: base64.urlsafe_b64encode(x).decode().rstrip("=")
print(b(blob), b(key))
PY
)"
[ -n "$CIPHERTEXT" ] || fail "could not encrypt (is 'cryptography' installed? pip install cryptography)"
pass "encrypted client-side (${#CIPHERTEXT} bytes of ciphertext)"
# --- create ----------------------------------------------------------------
CREATE=$(curl -sS -X POST "$BASE/api/secrets" -H 'Content-Type: application/json' \
-d "{\"ciphertext\":\"$CIPHERTEXT\",\"ttl_seconds\":900}")
ID=$(printf '%s' "$CREATE" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("id",""))')
[ -n "$ID" ] || fail "create returned no id: $CREATE"
pass "created — id is ${#ID} chars"
# --- the previewer property, checked BEFORE revealing ----------------------
for _ in 1 2 3; do
code=$(curl -sS -o /dev/null -w '%{http_code}' "$BASE/s/$ID")
[ "$code" = "200" ] || fail "GET /s/{id} returned $code"
done
pass "GET on the reveal page x3 — Slack/Outlook previews are harmless"
# --- reveal once and decrypt ----------------------------------------------
REVEAL=$(curl -sS -X POST "$BASE/api/secrets/$ID/reveal")
GOT_CT=$(printf '%s' "$REVEAL" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("ciphertext",""))')
[ -n "$GOT_CT" ] || fail "reveal returned no ciphertext: $REVEAL"
DECRYPTED=$(python3 - "$GOT_CT" "$KEY" <<'PY'
import base64, sys
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
u = lambda s: base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
blob, key = u(sys.argv[1]), u(sys.argv[2])
sys.stdout.write(AESGCM(key).decrypt(blob[:12], blob[12:], None).decode())
PY
)
[ "$DECRYPTED" = "$PLAINTEXT" ] || fail "decrypted to '$DECRYPTED', expected '$PLAINTEXT'"
pass "revealed and decrypted — round trip matches"
# --- and it is gone --------------------------------------------------------
code=$(curl -sS -o /tmp/hush-gone.$$ -w '%{http_code}' -X POST "$BASE/api/secrets/$ID/reveal")
[ "$code" = "410" ] || fail "second reveal returned $code, want 410"
GONE_BODY=$(cat /tmp/hush-gone.$$); rm -f /tmp/hush-gone.$$
pass "second reveal is 410 gone"
# --- missing and malformed are one response --------------------------------
MISSING=$(curl -sS -X POST "$BASE/api/secrets/$(python3 -c 'print("A"*43)')/reveal")
MALFORMED=$(curl -sS -X POST "$BASE/api/secrets/not-an-id/reveal")
for body in "$MISSING" "$MALFORMED"; do
norm=$(printf '%s' "$body" | python3 -c 'import json,sys; e=json.load(sys.stdin)["error"]; print(e["code"], e["message"])')
gone=$(printf '%s' "$GONE_BODY" | python3 -c 'import json,sys; e=json.load(sys.stdin)["error"]; print(e["code"], e["message"])')
[ "$norm" = "$gone" ] || fail "responses differ between causes: '$norm' vs '$gone'"
done
pass "revealed, expired, missing and malformed are one indistinguishable response"
# --- the server must refuse plaintext -------------------------------------
code=$(curl -sS -o /dev/null -w '%{http_code}' -X POST "$BASE/api/secrets" \
-H 'Content-Type: application/json' -d '{"secret":"hunter2"}')
[ "$code" != "201" ] || fail "the server ACCEPTED a plaintext field — the zero-knowledge claim is broken"
pass "a plaintext field is refused ($code)"
echo
printf '\033[32mall smoke checks passed\033[0m\n'

20
vendor/github.com/beorn7/perks/LICENSE generated vendored Normal file
View File

@ -0,0 +1,20 @@
Copyright (C) 2013 Blake Mizerany
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

2388
vendor/github.com/beorn7/perks/quantile/exampledata.txt generated vendored Normal file

File diff suppressed because it is too large Load Diff

316
vendor/github.com/beorn7/perks/quantile/stream.go generated vendored Normal file
View File

@ -0,0 +1,316 @@
// Package quantile computes approximate quantiles over an unbounded data
// stream within low memory and CPU bounds.
//
// A small amount of accuracy is traded to achieve the above properties.
//
// Multiple streams can be merged before calling Query to generate a single set
// of results. This is meaningful when the streams represent the same type of
// data. See Merge and Samples.
//
// For more detailed information about the algorithm used, see:
//
// Effective Computation of Biased Quantiles over Data Streams
//
// http://www.cs.rutgers.edu/~muthu/bquant.pdf
package quantile
import (
"math"
"sort"
)
// Sample holds an observed value and meta information for compression. JSON
// tags have been added for convenience.
type Sample struct {
Value float64 `json:",string"`
Width float64 `json:",string"`
Delta float64 `json:",string"`
}
// Samples represents a slice of samples. It implements sort.Interface.
type Samples []Sample
func (a Samples) Len() int { return len(a) }
func (a Samples) Less(i, j int) bool { return a[i].Value < a[j].Value }
func (a Samples) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
type invariant func(s *stream, r float64) float64
// NewLowBiased returns an initialized Stream for low-biased quantiles
// (e.g. 0.01, 0.1, 0.5) where the needed quantiles are not known a priori, but
// error guarantees can still be given even for the lower ranks of the data
// distribution.
//
// The provided epsilon is a relative error, i.e. the true quantile of a value
// returned by a query is guaranteed to be within (1±Epsilon)*Quantile.
//
// See http://www.cs.rutgers.edu/~muthu/bquant.pdf for time, space, and error
// properties.
func NewLowBiased(epsilon float64) *Stream {
ƒ := func(s *stream, r float64) float64 {
return 2 * epsilon * r
}
return newStream(ƒ)
}
// NewHighBiased returns an initialized Stream for high-biased quantiles
// (e.g. 0.01, 0.1, 0.5) where the needed quantiles are not known a priori, but
// error guarantees can still be given even for the higher ranks of the data
// distribution.
//
// The provided epsilon is a relative error, i.e. the true quantile of a value
// returned by a query is guaranteed to be within 1-(1±Epsilon)*(1-Quantile).
//
// See http://www.cs.rutgers.edu/~muthu/bquant.pdf for time, space, and error
// properties.
func NewHighBiased(epsilon float64) *Stream {
ƒ := func(s *stream, r float64) float64 {
return 2 * epsilon * (s.n - r)
}
return newStream(ƒ)
}
// NewTargeted returns an initialized Stream concerned with a particular set of
// quantile values that are supplied a priori. Knowing these a priori reduces
// space and computation time. The targets map maps the desired quantiles to
// their absolute errors, i.e. the true quantile of a value returned by a query
// is guaranteed to be within (Quantile±Epsilon).
//
// See http://www.cs.rutgers.edu/~muthu/bquant.pdf for time, space, and error properties.
func NewTargeted(targetMap map[float64]float64) *Stream {
// Convert map to slice to avoid slow iterations on a map.
// ƒ is called on the hot path, so converting the map to a slice
// beforehand results in significant CPU savings.
targets := targetMapToSlice(targetMap)
ƒ := func(s *stream, r float64) float64 {
var m = math.MaxFloat64
var f float64
for _, t := range targets {
if t.quantile*s.n <= r {
f = (2 * t.epsilon * r) / t.quantile
} else {
f = (2 * t.epsilon * (s.n - r)) / (1 - t.quantile)
}
if f < m {
m = f
}
}
return m
}
return newStream(ƒ)
}
type target struct {
quantile float64
epsilon float64
}
func targetMapToSlice(targetMap map[float64]float64) []target {
targets := make([]target, 0, len(targetMap))
for quantile, epsilon := range targetMap {
t := target{
quantile: quantile,
epsilon: epsilon,
}
targets = append(targets, t)
}
return targets
}
// Stream computes quantiles for a stream of float64s. It is not thread-safe by
// design. Take care when using across multiple goroutines.
type Stream struct {
*stream
b Samples
sorted bool
}
func newStream(ƒ invariant) *Stream {
x := &stream{ƒ: ƒ}
return &Stream{x, make(Samples, 0, 500), true}
}
// Insert inserts v into the stream.
func (s *Stream) Insert(v float64) {
s.insert(Sample{Value: v, Width: 1})
}
func (s *Stream) insert(sample Sample) {
s.b = append(s.b, sample)
s.sorted = false
if len(s.b) == cap(s.b) {
s.flush()
}
}
// Query returns the computed qth percentiles value. If s was created with
// NewTargeted, and q is not in the set of quantiles provided a priori, Query
// will return an unspecified result.
func (s *Stream) Query(q float64) float64 {
if !s.flushed() {
// Fast path when there hasn't been enough data for a flush;
// this also yields better accuracy for small sets of data.
l := len(s.b)
if l == 0 {
return 0
}
i := int(math.Ceil(float64(l) * q))
if i > 0 {
i -= 1
}
s.maybeSort()
return s.b[i].Value
}
s.flush()
return s.stream.query(q)
}
// Merge merges samples into the underlying streams samples. This is handy when
// merging multiple streams from separate threads, database shards, etc.
//
// ATTENTION: This method is broken and does not yield correct results. The
// underlying algorithm is not capable of merging streams correctly.
func (s *Stream) Merge(samples Samples) {
sort.Sort(samples)
s.stream.merge(samples)
}
// Reset reinitializes and clears the list reusing the samples buffer memory.
func (s *Stream) Reset() {
s.stream.reset()
s.b = s.b[:0]
}
// Samples returns stream samples held by s.
func (s *Stream) Samples() Samples {
if !s.flushed() {
return s.b
}
s.flush()
return s.stream.samples()
}
// Count returns the total number of samples observed in the stream
// since initialization.
func (s *Stream) Count() int {
return len(s.b) + s.stream.count()
}
func (s *Stream) flush() {
s.maybeSort()
s.stream.merge(s.b)
s.b = s.b[:0]
}
func (s *Stream) maybeSort() {
if !s.sorted {
s.sorted = true
sort.Sort(s.b)
}
}
func (s *Stream) flushed() bool {
return len(s.stream.l) > 0
}
type stream struct {
n float64
l []Sample
ƒ invariant
}
func (s *stream) reset() {
s.l = s.l[:0]
s.n = 0
}
func (s *stream) insert(v float64) {
s.merge(Samples{{v, 1, 0}})
}
func (s *stream) merge(samples Samples) {
// TODO(beorn7): This tries to merge not only individual samples, but
// whole summaries. The paper doesn't mention merging summaries at
// all. Unittests show that the merging is inaccurate. Find out how to
// do merges properly.
var r float64
i := 0
for _, sample := range samples {
for ; i < len(s.l); i++ {
c := s.l[i]
if c.Value > sample.Value {
// Insert at position i.
s.l = append(s.l, Sample{})
copy(s.l[i+1:], s.l[i:])
s.l[i] = Sample{
sample.Value,
sample.Width,
math.Max(sample.Delta, math.Floor(s.ƒ(s, r))-1),
// TODO(beorn7): How to calculate delta correctly?
}
i++
goto inserted
}
r += c.Width
}
s.l = append(s.l, Sample{sample.Value, sample.Width, 0})
i++
inserted:
s.n += sample.Width
r += sample.Width
}
s.compress()
}
func (s *stream) count() int {
return int(s.n)
}
func (s *stream) query(q float64) float64 {
t := math.Ceil(q * s.n)
t += math.Ceil(s.ƒ(s, t) / 2)
p := s.l[0]
var r float64
for _, c := range s.l[1:] {
r += p.Width
if r+c.Width+c.Delta > t {
return p.Value
}
p = c
}
return p.Value
}
func (s *stream) compress() {
if len(s.l) < 2 {
return
}
x := s.l[len(s.l)-1]
xi := len(s.l) - 1
r := s.n - 1 - x.Width
for i := len(s.l) - 2; i >= 0; i-- {
c := s.l[i]
if c.Width+x.Width+x.Delta <= s.ƒ(s, r) {
x.Width += c.Width
s.l[xi] = x
// Remove element at i.
copy(s.l[i:], s.l[i+1:])
s.l = s.l[:len(s.l)-1]
xi -= 1
} else {
x = c
xi = i
}
r -= c.Width
}
}
func (s *stream) samples() Samples {
samples := make(Samples, len(s.l))
copy(samples, s.l)
return samples
}

22
vendor/github.com/cespare/xxhash/v2/LICENSE.txt generated vendored Normal file
View File

@ -0,0 +1,22 @@
Copyright (c) 2016 Caleb Spare
MIT License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

74
vendor/github.com/cespare/xxhash/v2/README.md generated vendored Normal file
View File

@ -0,0 +1,74 @@
# xxhash
[![Go Reference](https://pkg.go.dev/badge/github.com/cespare/xxhash/v2.svg)](https://pkg.go.dev/github.com/cespare/xxhash/v2)
[![Test](https://github.com/cespare/xxhash/actions/workflows/test.yml/badge.svg)](https://github.com/cespare/xxhash/actions/workflows/test.yml)
xxhash is a Go implementation of the 64-bit [xxHash] algorithm, XXH64. This is a
high-quality hashing algorithm that is much faster than anything in the Go
standard library.
This package provides a straightforward API:
```
func Sum64(b []byte) uint64
func Sum64String(s string) uint64
type Digest struct{ ... }
func New() *Digest
```
The `Digest` type implements hash.Hash64. Its key methods are:
```
func (*Digest) Write([]byte) (int, error)
func (*Digest) WriteString(string) (int, error)
func (*Digest) Sum64() uint64
```
The package is written with optimized pure Go and also contains even faster
assembly implementations for amd64 and arm64. If desired, the `purego` build tag
opts into using the Go code even on those architectures.
[xxHash]: http://cyan4973.github.io/xxHash/
## Compatibility
This package is in a module and the latest code is in version 2 of the module.
You need a version of Go with at least "minimal module compatibility" to use
github.com/cespare/xxhash/v2:
* 1.9.7+ for Go 1.9
* 1.10.3+ for Go 1.10
* Go 1.11 or later
I recommend using the latest release of Go.
## Benchmarks
Here are some quick benchmarks comparing the pure-Go and assembly
implementations of Sum64.
| input size | purego | asm |
| ---------- | --------- | --------- |
| 4 B | 1.3 GB/s | 1.2 GB/s |
| 16 B | 2.9 GB/s | 3.5 GB/s |
| 100 B | 6.9 GB/s | 8.1 GB/s |
| 4 KB | 11.7 GB/s | 16.7 GB/s |
| 10 MB | 12.0 GB/s | 17.3 GB/s |
These numbers were generated on Ubuntu 20.04 with an Intel Xeon Platinum 8252C
CPU using the following commands under Go 1.19.2:
```
benchstat <(go test -tags purego -benchtime 500ms -count 15 -bench 'Sum64$')
benchstat <(go test -benchtime 500ms -count 15 -bench 'Sum64$')
```
## Projects using this package
- [InfluxDB](https://github.com/influxdata/influxdb)
- [Prometheus](https://github.com/prometheus/prometheus)
- [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics)
- [FreeCache](https://github.com/coocood/freecache)
- [FastCache](https://github.com/VictoriaMetrics/fastcache)
- [Ristretto](https://github.com/dgraph-io/ristretto)
- [Badger](https://github.com/dgraph-io/badger)

10
vendor/github.com/cespare/xxhash/v2/testall.sh generated vendored Normal file
View File

@ -0,0 +1,10 @@
#!/bin/bash
set -eu -o pipefail
# Small convenience script for running the tests with various combinations of
# arch/tags. This assumes we're running on amd64 and have qemu available.
go test ./...
go test -tags purego ./...
GOARCH=arm64 go test
GOARCH=arm64 go test -tags purego

243
vendor/github.com/cespare/xxhash/v2/xxhash.go generated vendored Normal file
View File

@ -0,0 +1,243 @@
// Package xxhash implements the 64-bit variant of xxHash (XXH64) as described
// at http://cyan4973.github.io/xxHash/.
package xxhash
import (
"encoding/binary"
"errors"
"math/bits"
)
const (
prime1 uint64 = 11400714785074694791
prime2 uint64 = 14029467366897019727
prime3 uint64 = 1609587929392839161
prime4 uint64 = 9650029242287828579
prime5 uint64 = 2870177450012600261
)
// Store the primes in an array as well.
//
// The consts are used when possible in Go code to avoid MOVs but we need a
// contiguous array for the assembly code.
var primes = [...]uint64{prime1, prime2, prime3, prime4, prime5}
// Digest implements hash.Hash64.
//
// Note that a zero-valued Digest is not ready to receive writes.
// Call Reset or create a Digest using New before calling other methods.
type Digest struct {
v1 uint64
v2 uint64
v3 uint64
v4 uint64
total uint64
mem [32]byte
n int // how much of mem is used
}
// New creates a new Digest with a zero seed.
func New() *Digest {
return NewWithSeed(0)
}
// NewWithSeed creates a new Digest with the given seed.
func NewWithSeed(seed uint64) *Digest {
var d Digest
d.ResetWithSeed(seed)
return &d
}
// Reset clears the Digest's state so that it can be reused.
// It uses a seed value of zero.
func (d *Digest) Reset() {
d.ResetWithSeed(0)
}
// ResetWithSeed clears the Digest's state so that it can be reused.
// It uses the given seed to initialize the state.
func (d *Digest) ResetWithSeed(seed uint64) {
d.v1 = seed + prime1 + prime2
d.v2 = seed + prime2
d.v3 = seed
d.v4 = seed - prime1
d.total = 0
d.n = 0
}
// Size always returns 8 bytes.
func (d *Digest) Size() int { return 8 }
// BlockSize always returns 32 bytes.
func (d *Digest) BlockSize() int { return 32 }
// Write adds more data to d. It always returns len(b), nil.
func (d *Digest) Write(b []byte) (n int, err error) {
n = len(b)
d.total += uint64(n)
memleft := d.mem[d.n&(len(d.mem)-1):]
if d.n+n < 32 {
// This new data doesn't even fill the current block.
copy(memleft, b)
d.n += n
return
}
if d.n > 0 {
// Finish off the partial block.
c := copy(memleft, b)
d.v1 = round(d.v1, u64(d.mem[0:8]))
d.v2 = round(d.v2, u64(d.mem[8:16]))
d.v3 = round(d.v3, u64(d.mem[16:24]))
d.v4 = round(d.v4, u64(d.mem[24:32]))
b = b[c:]
d.n = 0
}
if len(b) >= 32 {
// One or more full blocks left.
nw := writeBlocks(d, b)
b = b[nw:]
}
// Store any remaining partial block.
copy(d.mem[:], b)
d.n = len(b)
return
}
// Sum appends the current hash to b and returns the resulting slice.
func (d *Digest) Sum(b []byte) []byte {
s := d.Sum64()
return append(
b,
byte(s>>56),
byte(s>>48),
byte(s>>40),
byte(s>>32),
byte(s>>24),
byte(s>>16),
byte(s>>8),
byte(s),
)
}
// Sum64 returns the current hash.
func (d *Digest) Sum64() uint64 {
var h uint64
if d.total >= 32 {
v1, v2, v3, v4 := d.v1, d.v2, d.v3, d.v4
h = rol1(v1) + rol7(v2) + rol12(v3) + rol18(v4)
h = mergeRound(h, v1)
h = mergeRound(h, v2)
h = mergeRound(h, v3)
h = mergeRound(h, v4)
} else {
h = d.v3 + prime5
}
h += d.total
b := d.mem[:d.n&(len(d.mem)-1)]
for ; len(b) >= 8; b = b[8:] {
k1 := round(0, u64(b[:8]))
h ^= k1
h = rol27(h)*prime1 + prime4
}
if len(b) >= 4 {
h ^= uint64(u32(b[:4])) * prime1
h = rol23(h)*prime2 + prime3
b = b[4:]
}
for ; len(b) > 0; b = b[1:] {
h ^= uint64(b[0]) * prime5
h = rol11(h) * prime1
}
h ^= h >> 33
h *= prime2
h ^= h >> 29
h *= prime3
h ^= h >> 32
return h
}
const (
magic = "xxh\x06"
marshaledSize = len(magic) + 8*5 + 32
)
// MarshalBinary implements the encoding.BinaryMarshaler interface.
func (d *Digest) MarshalBinary() ([]byte, error) {
b := make([]byte, 0, marshaledSize)
b = append(b, magic...)
b = appendUint64(b, d.v1)
b = appendUint64(b, d.v2)
b = appendUint64(b, d.v3)
b = appendUint64(b, d.v4)
b = appendUint64(b, d.total)
b = append(b, d.mem[:d.n]...)
b = b[:len(b)+len(d.mem)-d.n]
return b, nil
}
// UnmarshalBinary implements the encoding.BinaryUnmarshaler interface.
func (d *Digest) UnmarshalBinary(b []byte) error {
if len(b) < len(magic) || string(b[:len(magic)]) != magic {
return errors.New("xxhash: invalid hash state identifier")
}
if len(b) != marshaledSize {
return errors.New("xxhash: invalid hash state size")
}
b = b[len(magic):]
b, d.v1 = consumeUint64(b)
b, d.v2 = consumeUint64(b)
b, d.v3 = consumeUint64(b)
b, d.v4 = consumeUint64(b)
b, d.total = consumeUint64(b)
copy(d.mem[:], b)
d.n = int(d.total % uint64(len(d.mem)))
return nil
}
func appendUint64(b []byte, x uint64) []byte {
var a [8]byte
binary.LittleEndian.PutUint64(a[:], x)
return append(b, a[:]...)
}
func consumeUint64(b []byte) ([]byte, uint64) {
x := u64(b)
return b[8:], x
}
func u64(b []byte) uint64 { return binary.LittleEndian.Uint64(b) }
func u32(b []byte) uint32 { return binary.LittleEndian.Uint32(b) }
func round(acc, input uint64) uint64 {
acc += input * prime2
acc = rol31(acc)
acc *= prime1
return acc
}
func mergeRound(acc, val uint64) uint64 {
val = round(0, val)
acc ^= val
acc = acc*prime1 + prime4
return acc
}
func rol1(x uint64) uint64 { return bits.RotateLeft64(x, 1) }
func rol7(x uint64) uint64 { return bits.RotateLeft64(x, 7) }
func rol11(x uint64) uint64 { return bits.RotateLeft64(x, 11) }
func rol12(x uint64) uint64 { return bits.RotateLeft64(x, 12) }
func rol18(x uint64) uint64 { return bits.RotateLeft64(x, 18) }
func rol23(x uint64) uint64 { return bits.RotateLeft64(x, 23) }
func rol27(x uint64) uint64 { return bits.RotateLeft64(x, 27) }
func rol31(x uint64) uint64 { return bits.RotateLeft64(x, 31) }

209
vendor/github.com/cespare/xxhash/v2/xxhash_amd64.s generated vendored Normal file
View File

@ -0,0 +1,209 @@
//go:build !appengine && gc && !purego
// +build !appengine
// +build gc
// +build !purego
#include "textflag.h"
// Registers:
#define h AX
#define d AX
#define p SI // pointer to advance through b
#define n DX
#define end BX // loop end
#define v1 R8
#define v2 R9
#define v3 R10
#define v4 R11
#define x R12
#define prime1 R13
#define prime2 R14
#define prime4 DI
#define round(acc, x) \
IMULQ prime2, x \
ADDQ x, acc \
ROLQ $31, acc \
IMULQ prime1, acc
// round0 performs the operation x = round(0, x).
#define round0(x) \
IMULQ prime2, x \
ROLQ $31, x \
IMULQ prime1, x
// mergeRound applies a merge round on the two registers acc and x.
// It assumes that prime1, prime2, and prime4 have been loaded.
#define mergeRound(acc, x) \
round0(x) \
XORQ x, acc \
IMULQ prime1, acc \
ADDQ prime4, acc
// blockLoop processes as many 32-byte blocks as possible,
// updating v1, v2, v3, and v4. It assumes that there is at least one block
// to process.
#define blockLoop() \
loop: \
MOVQ +0(p), x \
round(v1, x) \
MOVQ +8(p), x \
round(v2, x) \
MOVQ +16(p), x \
round(v3, x) \
MOVQ +24(p), x \
round(v4, x) \
ADDQ $32, p \
CMPQ p, end \
JLE loop
// func Sum64(b []byte) uint64
TEXT ·Sum64(SB), NOSPLIT|NOFRAME, $0-32
// Load fixed primes.
MOVQ ·primes+0(SB), prime1
MOVQ ·primes+8(SB), prime2
MOVQ ·primes+24(SB), prime4
// Load slice.
MOVQ b_base+0(FP), p
MOVQ b_len+8(FP), n
LEAQ (p)(n*1), end
// The first loop limit will be len(b)-32.
SUBQ $32, end
// Check whether we have at least one block.
CMPQ n, $32
JLT noBlocks
// Set up initial state (v1, v2, v3, v4).
MOVQ prime1, v1
ADDQ prime2, v1
MOVQ prime2, v2
XORQ v3, v3
XORQ v4, v4
SUBQ prime1, v4
blockLoop()
MOVQ v1, h
ROLQ $1, h
MOVQ v2, x
ROLQ $7, x
ADDQ x, h
MOVQ v3, x
ROLQ $12, x
ADDQ x, h
MOVQ v4, x
ROLQ $18, x
ADDQ x, h
mergeRound(h, v1)
mergeRound(h, v2)
mergeRound(h, v3)
mergeRound(h, v4)
JMP afterBlocks
noBlocks:
MOVQ ·primes+32(SB), h
afterBlocks:
ADDQ n, h
ADDQ $24, end
CMPQ p, end
JG try4
loop8:
MOVQ (p), x
ADDQ $8, p
round0(x)
XORQ x, h
ROLQ $27, h
IMULQ prime1, h
ADDQ prime4, h
CMPQ p, end
JLE loop8
try4:
ADDQ $4, end
CMPQ p, end
JG try1
MOVL (p), x
ADDQ $4, p
IMULQ prime1, x
XORQ x, h
ROLQ $23, h
IMULQ prime2, h
ADDQ ·primes+16(SB), h
try1:
ADDQ $4, end
CMPQ p, end
JGE finalize
loop1:
MOVBQZX (p), x
ADDQ $1, p
IMULQ ·primes+32(SB), x
XORQ x, h
ROLQ $11, h
IMULQ prime1, h
CMPQ p, end
JL loop1
finalize:
MOVQ h, x
SHRQ $33, x
XORQ x, h
IMULQ prime2, h
MOVQ h, x
SHRQ $29, x
XORQ x, h
IMULQ ·primes+16(SB), h
MOVQ h, x
SHRQ $32, x
XORQ x, h
MOVQ h, ret+24(FP)
RET
// func writeBlocks(d *Digest, b []byte) int
TEXT ·writeBlocks(SB), NOSPLIT|NOFRAME, $0-40
// Load fixed primes needed for round.
MOVQ ·primes+0(SB), prime1
MOVQ ·primes+8(SB), prime2
// Load slice.
MOVQ b_base+8(FP), p
MOVQ b_len+16(FP), n
LEAQ (p)(n*1), end
SUBQ $32, end
// Load vN from d.
MOVQ s+0(FP), d
MOVQ 0(d), v1
MOVQ 8(d), v2
MOVQ 16(d), v3
MOVQ 24(d), v4
// We don't need to check the loop condition here; this function is
// always called with at least one block of data to process.
blockLoop()
// Copy vN back to d.
MOVQ v1, 0(d)
MOVQ v2, 8(d)
MOVQ v3, 16(d)
MOVQ v4, 24(d)
// The number of bytes written is p minus the old base pointer.
SUBQ b_base+8(FP), p
MOVQ p, ret+32(FP)
RET

183
vendor/github.com/cespare/xxhash/v2/xxhash_arm64.s generated vendored Normal file
View File

@ -0,0 +1,183 @@
//go:build !appengine && gc && !purego
// +build !appengine
// +build gc
// +build !purego
#include "textflag.h"
// Registers:
#define digest R1
#define h R2 // return value
#define p R3 // input pointer
#define n R4 // input length
#define nblocks R5 // n / 32
#define prime1 R7
#define prime2 R8
#define prime3 R9
#define prime4 R10
#define prime5 R11
#define v1 R12
#define v2 R13
#define v3 R14
#define v4 R15
#define x1 R20
#define x2 R21
#define x3 R22
#define x4 R23
#define round(acc, x) \
MADD prime2, acc, x, acc \
ROR $64-31, acc \
MUL prime1, acc
// round0 performs the operation x = round(0, x).
#define round0(x) \
MUL prime2, x \
ROR $64-31, x \
MUL prime1, x
#define mergeRound(acc, x) \
round0(x) \
EOR x, acc \
MADD acc, prime4, prime1, acc
// blockLoop processes as many 32-byte blocks as possible,
// updating v1, v2, v3, and v4. It assumes that n >= 32.
#define blockLoop() \
LSR $5, n, nblocks \
PCALIGN $16 \
loop: \
LDP.P 16(p), (x1, x2) \
LDP.P 16(p), (x3, x4) \
round(v1, x1) \
round(v2, x2) \
round(v3, x3) \
round(v4, x4) \
SUB $1, nblocks \
CBNZ nblocks, loop
// func Sum64(b []byte) uint64
TEXT ·Sum64(SB), NOSPLIT|NOFRAME, $0-32
LDP b_base+0(FP), (p, n)
LDP ·primes+0(SB), (prime1, prime2)
LDP ·primes+16(SB), (prime3, prime4)
MOVD ·primes+32(SB), prime5
CMP $32, n
CSEL LT, prime5, ZR, h // if n < 32 { h = prime5 } else { h = 0 }
BLT afterLoop
ADD prime1, prime2, v1
MOVD prime2, v2
MOVD $0, v3
NEG prime1, v4
blockLoop()
ROR $64-1, v1, x1
ROR $64-7, v2, x2
ADD x1, x2
ROR $64-12, v3, x3
ROR $64-18, v4, x4
ADD x3, x4
ADD x2, x4, h
mergeRound(h, v1)
mergeRound(h, v2)
mergeRound(h, v3)
mergeRound(h, v4)
afterLoop:
ADD n, h
TBZ $4, n, try8
LDP.P 16(p), (x1, x2)
round0(x1)
// NOTE: here and below, sequencing the EOR after the ROR (using a
// rotated register) is worth a small but measurable speedup for small
// inputs.
ROR $64-27, h
EOR x1 @> 64-27, h, h
MADD h, prime4, prime1, h
round0(x2)
ROR $64-27, h
EOR x2 @> 64-27, h, h
MADD h, prime4, prime1, h
try8:
TBZ $3, n, try4
MOVD.P 8(p), x1
round0(x1)
ROR $64-27, h
EOR x1 @> 64-27, h, h
MADD h, prime4, prime1, h
try4:
TBZ $2, n, try2
MOVWU.P 4(p), x2
MUL prime1, x2
ROR $64-23, h
EOR x2 @> 64-23, h, h
MADD h, prime3, prime2, h
try2:
TBZ $1, n, try1
MOVHU.P 2(p), x3
AND $255, x3, x1
LSR $8, x3, x2
MUL prime5, x1
ROR $64-11, h
EOR x1 @> 64-11, h, h
MUL prime1, h
MUL prime5, x2
ROR $64-11, h
EOR x2 @> 64-11, h, h
MUL prime1, h
try1:
TBZ $0, n, finalize
MOVBU (p), x4
MUL prime5, x4
ROR $64-11, h
EOR x4 @> 64-11, h, h
MUL prime1, h
finalize:
EOR h >> 33, h
MUL prime2, h
EOR h >> 29, h
MUL prime3, h
EOR h >> 32, h
MOVD h, ret+24(FP)
RET
// func writeBlocks(d *Digest, b []byte) int
TEXT ·writeBlocks(SB), NOSPLIT|NOFRAME, $0-40
LDP ·primes+0(SB), (prime1, prime2)
// Load state. Assume v[1-4] are stored contiguously.
MOVD d+0(FP), digest
LDP 0(digest), (v1, v2)
LDP 16(digest), (v3, v4)
LDP b_base+8(FP), (p, n)
blockLoop()
// Store updated state.
STP (v1, v2), 0(digest)
STP (v3, v4), 16(digest)
BIC $31, n
MOVD n, ret+32(FP)
RET

15
vendor/github.com/cespare/xxhash/v2/xxhash_asm.go generated vendored Normal file
View File

@ -0,0 +1,15 @@
//go:build (amd64 || arm64) && !appengine && gc && !purego
// +build amd64 arm64
// +build !appengine
// +build gc
// +build !purego
package xxhash
// Sum64 computes the 64-bit xxHash digest of b with a zero seed.
//
//go:noescape
func Sum64(b []byte) uint64
//go:noescape
func writeBlocks(d *Digest, b []byte) int

76
vendor/github.com/cespare/xxhash/v2/xxhash_other.go generated vendored Normal file
View File

@ -0,0 +1,76 @@
//go:build (!amd64 && !arm64) || appengine || !gc || purego
// +build !amd64,!arm64 appengine !gc purego
package xxhash
// Sum64 computes the 64-bit xxHash digest of b with a zero seed.
func Sum64(b []byte) uint64 {
// A simpler version would be
// d := New()
// d.Write(b)
// return d.Sum64()
// but this is faster, particularly for small inputs.
n := len(b)
var h uint64
if n >= 32 {
v1 := primes[0] + prime2
v2 := prime2
v3 := uint64(0)
v4 := -primes[0]
for len(b) >= 32 {
v1 = round(v1, u64(b[0:8:len(b)]))
v2 = round(v2, u64(b[8:16:len(b)]))
v3 = round(v3, u64(b[16:24:len(b)]))
v4 = round(v4, u64(b[24:32:len(b)]))
b = b[32:len(b):len(b)]
}
h = rol1(v1) + rol7(v2) + rol12(v3) + rol18(v4)
h = mergeRound(h, v1)
h = mergeRound(h, v2)
h = mergeRound(h, v3)
h = mergeRound(h, v4)
} else {
h = prime5
}
h += uint64(n)
for ; len(b) >= 8; b = b[8:] {
k1 := round(0, u64(b[:8]))
h ^= k1
h = rol27(h)*prime1 + prime4
}
if len(b) >= 4 {
h ^= uint64(u32(b[:4])) * prime1
h = rol23(h)*prime2 + prime3
b = b[4:]
}
for ; len(b) > 0; b = b[1:] {
h ^= uint64(b[0]) * prime5
h = rol11(h) * prime1
}
h ^= h >> 33
h *= prime2
h ^= h >> 29
h *= prime3
h ^= h >> 32
return h
}
func writeBlocks(d *Digest, b []byte) int {
v1, v2, v3, v4 := d.v1, d.v2, d.v3, d.v4
n := len(b)
for len(b) >= 32 {
v1 = round(v1, u64(b[0:8:len(b)]))
v2 = round(v2, u64(b[8:16:len(b)]))
v3 = round(v3, u64(b[16:24:len(b)]))
v4 = round(v4, u64(b[24:32:len(b)]))
b = b[32:len(b):len(b)]
}
d.v1, d.v2, d.v3, d.v4 = v1, v2, v3, v4
return n - len(b)
}

16
vendor/github.com/cespare/xxhash/v2/xxhash_safe.go generated vendored Normal file
View File

@ -0,0 +1,16 @@
//go:build appengine
// +build appengine
// This file contains the safe implementations of otherwise unsafe-using code.
package xxhash
// Sum64String computes the 64-bit xxHash digest of s with a zero seed.
func Sum64String(s string) uint64 {
return Sum64([]byte(s))
}
// WriteString adds more data to d. It always returns len(s), nil.
func (d *Digest) WriteString(s string) (n int, err error) {
return d.Write([]byte(s))
}

58
vendor/github.com/cespare/xxhash/v2/xxhash_unsafe.go generated vendored Normal file
View File

@ -0,0 +1,58 @@
//go:build !appengine
// +build !appengine
// This file encapsulates usage of unsafe.
// xxhash_safe.go contains the safe implementations.
package xxhash
import (
"unsafe"
)
// In the future it's possible that compiler optimizations will make these
// XxxString functions unnecessary by realizing that calls such as
// Sum64([]byte(s)) don't need to copy s. See https://go.dev/issue/2205.
// If that happens, even if we keep these functions they can be replaced with
// the trivial safe code.
// NOTE: The usual way of doing an unsafe string-to-[]byte conversion is:
//
// var b []byte
// bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))
// bh.Data = (*reflect.StringHeader)(unsafe.Pointer(&s)).Data
// bh.Len = len(s)
// bh.Cap = len(s)
//
// Unfortunately, as of Go 1.15.3 the inliner's cost model assigns a high enough
// weight to this sequence of expressions that any function that uses it will
// not be inlined. Instead, the functions below use a different unsafe
// conversion designed to minimize the inliner weight and allow both to be
// inlined. There is also a test (TestInlining) which verifies that these are
// inlined.
//
// See https://github.com/golang/go/issues/42739 for discussion.
// Sum64String computes the 64-bit xxHash digest of s with a zero seed.
// It may be faster than Sum64([]byte(s)) by avoiding a copy.
func Sum64String(s string) uint64 {
b := *(*[]byte)(unsafe.Pointer(&sliceHeader{s, len(s)}))
return Sum64(b)
}
// WriteString adds more data to d. It always returns len(s), nil.
// It may be faster than Write([]byte(s)) by avoiding a copy.
func (d *Digest) WriteString(s string) (n int, err error) {
d.Write(*(*[]byte)(unsafe.Pointer(&sliceHeader{s, len(s)})))
// d.Write always returns len(s), nil.
// Ignoring the return output and returning these fixed values buys a
// savings of 6 in the inliner's cost model.
return len(s), nil
}
// sliceHeader is similar to reflect.SliceHeader, but it assumes that the layout
// of the first two words is the same as the layout of a string.
type sliceHeader struct {
s string
cap int
}

31
vendor/github.com/munnerz/goautoneg/LICENSE generated vendored Normal file
View File

@ -0,0 +1,31 @@
Copyright (c) 2011, Open Knowledge Foundation Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
Neither the name of the Open Knowledge Foundation Ltd. nor the
names of its contributors may be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

13
vendor/github.com/munnerz/goautoneg/Makefile generated vendored Normal file
View File

@ -0,0 +1,13 @@
include $(GOROOT)/src/Make.inc
TARG=bitbucket.org/ww/goautoneg
GOFILES=autoneg.go
include $(GOROOT)/src/Make.pkg
format:
gofmt -w *.go
docs:
gomake clean
godoc ${TARG} > README.txt

67
vendor/github.com/munnerz/goautoneg/README.txt generated vendored Normal file
View File

@ -0,0 +1,67 @@
PACKAGE
package goautoneg
import "bitbucket.org/ww/goautoneg"
HTTP Content-Type Autonegotiation.
The functions in this package implement the behaviour specified in
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Copyright (c) 2011, Open Knowledge Foundation Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
Neither the name of the Open Knowledge Foundation Ltd. nor the
names of its contributors may be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
FUNCTIONS
func Negotiate(header string, alternatives []string) (content_type string)
Negotiate the most appropriate content_type given the accept header
and a list of alternatives.
func ParseAccept(header string) (accept []Accept)
Parse an Accept Header string returning a sorted list
of clauses
TYPES
type Accept struct {
Type, SubType string
Q float32
Params map[string]string
}
Structure to represent a clause in an HTTP Accept Header
SUBDIRECTORIES
.hg

189
vendor/github.com/munnerz/goautoneg/autoneg.go generated vendored Normal file
View File

@ -0,0 +1,189 @@
/*
HTTP Content-Type Autonegotiation.
The functions in this package implement the behaviour specified in
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Copyright (c) 2011, Open Knowledge Foundation Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
Neither the name of the Open Knowledge Foundation Ltd. nor the
names of its contributors may be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
package goautoneg
import (
"sort"
"strconv"
"strings"
)
// Structure to represent a clause in an HTTP Accept Header
type Accept struct {
Type, SubType string
Q float64
Params map[string]string
}
// acceptSlice is defined to implement sort interface.
type acceptSlice []Accept
func (slice acceptSlice) Len() int {
return len(slice)
}
func (slice acceptSlice) Less(i, j int) bool {
ai, aj := slice[i], slice[j]
if ai.Q > aj.Q {
return true
}
if ai.Type != "*" && aj.Type == "*" {
return true
}
if ai.SubType != "*" && aj.SubType == "*" {
return true
}
return false
}
func (slice acceptSlice) Swap(i, j int) {
slice[i], slice[j] = slice[j], slice[i]
}
func stringTrimSpaceCutset(r rune) bool {
return r == ' '
}
func nextSplitElement(s, sep string) (item string, remaining string) {
if index := strings.Index(s, sep); index != -1 {
return s[:index], s[index+1:]
}
return s, ""
}
// Parse an Accept Header string returning a sorted list
// of clauses
func ParseAccept(header string) acceptSlice {
partsCount := 0
remaining := header
for len(remaining) > 0 {
partsCount++
_, remaining = nextSplitElement(remaining, ",")
}
accept := make(acceptSlice, 0, partsCount)
remaining = header
var part string
for len(remaining) > 0 {
part, remaining = nextSplitElement(remaining, ",")
part = strings.TrimFunc(part, stringTrimSpaceCutset)
a := Accept{
Q: 1.0,
}
sp, remainingPart := nextSplitElement(part, ";")
sp0, spRemaining := nextSplitElement(sp, "/")
a.Type = strings.TrimFunc(sp0, stringTrimSpaceCutset)
switch {
case len(spRemaining) == 0:
if a.Type == "*" {
a.SubType = "*"
} else {
continue
}
default:
var sp1 string
sp1, spRemaining = nextSplitElement(spRemaining, "/")
if len(spRemaining) > 0 {
continue
}
a.SubType = strings.TrimFunc(sp1, stringTrimSpaceCutset)
}
if len(remainingPart) == 0 {
accept = append(accept, a)
continue
}
a.Params = make(map[string]string)
for len(remainingPart) > 0 {
sp, remainingPart = nextSplitElement(remainingPart, ";")
sp0, spRemaining = nextSplitElement(sp, "=")
if len(spRemaining) == 0 {
continue
}
var sp1 string
sp1, spRemaining = nextSplitElement(spRemaining, "=")
if len(spRemaining) != 0 {
continue
}
token := strings.TrimFunc(sp0, stringTrimSpaceCutset)
if token == "q" {
a.Q, _ = strconv.ParseFloat(sp1, 32)
} else {
a.Params[token] = strings.TrimFunc(sp1, stringTrimSpaceCutset)
}
}
accept = append(accept, a)
}
sort.Sort(accept)
return accept
}
// Negotiate the most appropriate content_type given the accept header
// and a list of alternatives.
func Negotiate(header string, alternatives []string) (content_type string) {
asp := make([][]string, 0, len(alternatives))
for _, ctype := range alternatives {
asp = append(asp, strings.SplitN(ctype, "/", 2))
}
for _, clause := range ParseAccept(header) {
for i, ctsp := range asp {
if clause.Type == ctsp[0] && clause.SubType == ctsp[1] {
content_type = alternatives[i]
return
}
if clause.Type == ctsp[0] && clause.SubType == "*" {
content_type = alternatives[i]
return
}
if clause.Type == "*" && clause.SubType == "*" {
content_type = alternatives[i]
return
}
}
}
return
}

393
vendor/github.com/orchard9/go-chassis/chassis/app.go generated vendored Normal file
View File

@ -0,0 +1,393 @@
// Package chassis is the shared HTTP framework every service surface is built
// on (the API service, any second service, the worker's health endpoint). It
// owns the edge concerns so handlers stay thin: routing (stdlib ServeMux), the
// request Context + JSON envelope, the error model, the edge middleware chain
// (recover, request-id, structured logging, RED metrics, security headers,
// CORS), pluggable health probes, /metrics, an auth seam, and two-phase
// graceful shutdown. See patterns/go-chassis.md.
package chassis
import (
"context"
"errors"
"fmt"
"log/slog"
"net/http"
"runtime/debug"
"sync"
"sync/atomic"
"time"
"github.com/prometheus/client_golang/prometheus"
)
// Config is the chassis runtime configuration. Zero values get safe defaults
// (withDefaults); Service + Addr are the only fields a caller must set.
type Config struct {
Service string // closed-enum service name (metrics/logs)
Env string // dev | staging | prod
Addr string // listen address, e.g. ":16150"
AllowOrigins []string // exact-match CORS allowlist for browser UIs (empty = none)
MaxBodyBytes int64 // request body cap (Bind) — default 1 MiB
RequestTimeout time.Duration // per-request context deadline — default 15s
ReadyTimeout time.Duration // readiness check budget — default 2s
DrainDelay time.Duration // phase-1 drain wait before Shutdown — default 5s
ShutdownTimeout time.Duration // phase-2 in-flight drain — default 25s
// Collectors are service-owned Prometheus collectors registered on the
// app's private registry alongside the RED/Go/process ones, so /metrics is
// one scrape and there is no package-level default registry to collide in.
// Composition roots pass domain metrics here.
Collectors []prometheus.Collector
// Validator, if set, runs on every Context.Bind after JSON decode; a non-nil
// result becomes 422 Unprocessable. Wire shared/validate at the composition root.
Validator func(any) error
// EdgeMiddleware are extra http.Handler wrappers applied OUTERMOST (before
// secureHeaders) — e.g. a tracing span middleware from shared/tracing.
// Composition roots inject these; the chassis stays dependency-light.
EdgeMiddleware []func(http.Handler) http.Handler
}
// socketHeadroom is how far the socket deadlines outlive the per-request
// context deadline.
//
// ReadTimeout and WriteTimeout used to be hardcoded at 15s while
// RequestTimeout was configurable, so a surface that raised the request budget
// (reeld runs 120s so a worker can stream a rendered mp4 on the completion
// call) still had its socket cut at 15s. That reads as a proxy 502 rather than
// the board's own 413/504, which is a completely different bug to chase.
//
// The socket MUST outlive the context, never the reverse: when the context
// expires the handler returns and the error envelope is written on a socket
// that is still open. Cutting the socket first truncates the response mid-body.
const socketHeadroom = 30 * time.Second
func (c *Config) withDefaults() {
if c.MaxBodyBytes == 0 {
c.MaxBodyBytes = 1 << 20
}
if c.RequestTimeout == 0 {
c.RequestTimeout = 15 * time.Second
}
if c.ReadyTimeout == 0 {
c.ReadyTimeout = 2 * time.Second
}
if c.DrainDelay == 0 {
c.DrainDelay = 5 * time.Second
}
if c.ShutdownTimeout == 0 {
c.ShutdownTimeout = 25 * time.Second
}
}
// App is one HTTP surface: routes + health checks + background workers wired at
// the composition root, served by Run.
type App struct {
cfg Config
log *slog.Logger
mux *http.ServeMux
metrics *metrics
checks []namedCheck
bg []func(context.Context) error
routes []Route
draining atomic.Bool
// closing is closed when phase-1 drain starts. Request handlers that hold
// a connection open indefinitely (SSE) select on it so they end at the
// drain delay instead of stalling Shutdown for the full ShutdownTimeout.
closing chan struct{}
}
// Route is one registered endpoint. Exposed so a spec-drift test can reconcile
// the OpenAPI document against what the router actually serves — a hand-kept
// list of paths diverges from the code silently, which is the whole failure
// mode API docs have.
type Route struct {
Method string
Pattern string
}
// New builds an App and registers the always-on public routes: /metrics,
// /healthz (liveness), /readyz (readiness).
func New(cfg Config, log *slog.Logger) *App {
cfg.withDefaults()
a := &App{
cfg: cfg, log: log,
mux: http.NewServeMux(),
metrics: newMetrics(cfg.Service, cfg.Collectors...),
closing: make(chan struct{}),
}
a.mux.Handle("GET /metrics", a.metrics.handler())
a.mux.Handle("GET /healthz", a.toHTTP(a.handleLive))
a.mux.Handle("GET /readyz", a.toHTTP(a.handleReady))
return a
}
// toHTTP adapts a HandlerFunc to net/http, mapping a returned error to the
// JSON error envelope.
func (a *App) toHTTP(h HandlerFunc) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
c := &Context{
w: w, r: r,
maxBytes: a.cfg.MaxBodyBytes,
validate: a.cfg.Validator,
closing: a.closing,
}
if err := h(c); err != nil {
a.writeError(c, err)
}
})
}
// writeError logs the failure (5xx at error, 4xx at debug — both with the
// internal cause) and writes the client envelope, which never carries the cause
// or any secret.
func (a *App) writeError(c *Context, err error) {
e := asError(err)
log := c.Log()
// The cause rides on BOTH branches. Error.cause is documented as
// "logged server-side", and dropping it on 4xx made that false for every
// cause the framework attaches — Bind attaches one on 400, 413 and 422 and
// nothing else ever sees it. The visible symptom: a rejected body logged as
// bare `bad_request` with no field name, so the same unexplained-rejection
// hunt WorkerEligible's reason string exists to prevent on the lease path.
cause := e.Msg
if e.cause != nil {
cause = e.cause.Error()
}
if e.Status >= http.StatusInternalServerError {
log.Error("request.error", "category", "request",
"error_type", e.Code, "http_status", e.Status, "error_msg", cause)
} else {
log.Debug("request.rejected", "category", "request",
"error_type", e.Code, "http_status", e.Status, "error_msg", cause)
}
// request_id rides in the body, not just the X-Request-Id header. A browser
// on a cross-origin deployment cannot read a response header unless it is
// explicitly exposed, so a body without it leaves the user with an error and
// nothing to quote to support. The Rust track's error.rs declares this exact
// triple as the contract for every surface; omitting it here made that claim
// false for half the projects this skill generates.
rid := c.w.Header().Get("X-Request-Id")
if rid == "" {
// Sentinel, matching the Rust track. A literal "-" is itself the signal
// that the request-id middleware is not wired, which an absent key is not.
rid = "-"
}
_ = c.JSON(e.Status, map[string]any{
"error": map[string]any{"code": e.Code, "message": e.Msg, "request_id": rid},
})
}
func (a *App) register(method, pattern string, h HandlerFunc, mw []Middleware) {
for i := len(mw) - 1; i >= 0; i-- { // mw[0] outermost
h = mw[i](h)
}
a.routes = append(a.routes, Route{Method: method, Pattern: pattern})
a.mux.Handle(method+" "+pattern, a.toHTTP(h))
}
// Routes returns every route registered through Get/Post/Handle/Route, in
// registration order. The always-on probes (/healthz, /readyz, /metrics) are
// registered directly on the mux and deliberately excluded — they are chassis
// infrastructure, not part of a service's documented API surface.
func (a *App) Routes() []Route {
out := make([]Route, len(a.routes))
copy(out, a.routes)
return out
}
// Get/Post/Handle register a top-level route with optional route middleware.
func (a *App) Get(pattern string, h HandlerFunc, mw ...Middleware) { a.register("GET", pattern, h, mw) }
func (a *App) Post(pattern string, h HandlerFunc, mw ...Middleware) {
a.register("POST", pattern, h, mw)
}
func (a *App) Handle(method, pattern string, h HandlerFunc, mw ...Middleware) {
a.register(method, pattern, h, mw)
}
// Route groups routes under a path prefix with shared middleware (e.g. auth).
func (a *App) Route(prefix string, fn func(r *Router)) { fn(&Router{app: a, prefix: prefix}) }
// Router registers routes under a prefix, applying group middleware to each.
type Router struct {
app *App
prefix string
mw []Middleware
}
// Use adds middleware applied to every route registered on this Router.
func (r *Router) Use(mw ...Middleware) { r.mw = append(r.mw, mw...) }
func (r *Router) Get(pattern string, h HandlerFunc, mw ...Middleware) {
r.handle("GET", pattern, h, mw)
}
func (r *Router) Post(pattern string, h HandlerFunc, mw ...Middleware) {
r.handle("POST", pattern, h, mw)
}
func (r *Router) Delete(pattern string, h HandlerFunc, mw ...Middleware) {
r.handle("DELETE", pattern, h, mw)
}
// Handle registers any method under the group prefix.
func (r *Router) Handle(method, pattern string, h HandlerFunc, mw ...Middleware) {
r.handle(method, pattern, h, mw)
}
func (r *Router) handle(method, pattern string, h HandlerFunc, mw []Middleware) {
all := make([]Middleware, 0, len(r.mw)+len(mw))
all = append(all, r.mw...)
all = append(all, mw...)
r.app.register(method, r.prefix+pattern, h, all)
}
// Health registers a named readiness dependency check.
func (a *App) Health(name string, fn CheckFunc) {
a.checks = append(a.checks, namedCheck{name: name, fn: fn})
}
// Background registers a worker run with the server lifecycle; it receives a
// context cancelled on shutdown and Run waits for it to return.
func (a *App) Background(fn func(context.Context) error) { a.bg = append(a.bg, fn) }
// Handler returns the fully composed edge chain over the route mux — used by Run
// and available for in-process tests. Order (outermost first): any injected
// EdgeMiddleware (e.g. tracing), then secureHeaders, instrument, CORS.
func (a *App) Handler() http.Handler {
mw := make([]func(http.Handler) http.Handler, 0, len(a.cfg.EdgeMiddleware)+3)
mw = append(mw, a.cfg.EdgeMiddleware...)
mw = append(mw, a.secureHeaders, a.instrument, a.cors)
return chain(a.mux, mw...)
}
// Run starts the server and blocks until ctx is cancelled (SIGINT/SIGTERM) or
// the listener fails. Shutdown is two-phase: flip readiness to 503 so the load
// balancer drains the pod, wait DrainDelay, then Shutdown in-flight requests
// under ShutdownTimeout (DrainDelay + ShutdownTimeout MUST be < the k8s grace
// period). Liveness stays 200 throughout so the pod is drained, never killed.
func (a *App) Run(ctx context.Context) error {
// The socket deadlines track RequestTimeout, they are not independent
// knobs: see socketHeadroom. ReadTimeout has to clear a full body upload
// (reeld takes a rendered mp4 on the completion call) and WriteTimeout a
// full response, both of which are bounded by the request budget.
srv := &http.Server{
Addr: a.cfg.Addr,
Handler: a.Handler(),
ReadHeaderTimeout: 5 * time.Second, // Slowloris guard
ReadTimeout: a.cfg.RequestTimeout + socketHeadroom,
WriteTimeout: a.cfg.RequestTimeout + socketHeadroom,
IdleTimeout: 60 * time.Second,
MaxHeaderBytes: 1 << 20,
}
runCtx, cancel := context.WithCancel(ctx)
defer cancel()
var wg sync.WaitGroup
for _, fn := range a.bg {
wg.Add(1)
go func(fn func(context.Context) error) {
defer wg.Done()
a.superviseBackground(runCtx, fn)
}(fn)
}
errc := make(chan error, 1)
go func() {
a.log.Info("server.listening", "http_addr", a.cfg.Addr)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
errc <- err
}
}()
select {
case err := <-errc:
cancel()
wg.Wait()
return err
case <-runCtx.Done():
a.draining.Store(true) // phase 1: readiness -> 503, LB drains the pod
// Long-lived streams end here rather than at phase 2: an SSE client
// that reconnects during the drain delay lands on a pod that is still
// serving, instead of one that is 25s from closing under it.
close(a.closing)
a.log.Info("server.draining", "category", "shutdown", "drain_delay_ms", a.cfg.DrainDelay.Milliseconds())
time.Sleep(a.cfg.DrainDelay)
sctx, scancel := context.WithTimeout(context.Background(), a.cfg.ShutdownTimeout) // phase 2: drain in-flight
defer scancel()
err := srv.Shutdown(sctx)
cancel()
wg.Wait()
a.log.Info("server.stopped", "category", "shutdown")
return err
}
}
// superviseBackground runs one background worker, restarting it with backoff if
// it panics or returns an error.
//
// Without this a panic in an auxiliary worker takes the whole process down,
// including HTTP serving — so a bug in, say, artifact retention would stop the
// board handing out leases. That asymmetry is surprising, because request
// handlers already get panic recovery from the instrument middleware; this
// gives background work the same protection.
//
// Restarting rather than merely recovering matters just as much: a worker that
// dies quietly leaves its job undone forever with a green readiness probe, and
// nobody discovers retention stopped until a disk fills.
func (a *App) superviseBackground(ctx context.Context, fn func(context.Context) error) {
const (
minBackoff = time.Second
maxBackoff = time.Minute
)
backoff := minBackoff
for {
err, panicked := runBackgroundOnce(ctx, fn)
switch {
case ctx.Err() != nil:
// Shutdown. A worker returning on a cancelled context is the
// normal path, not a failure.
return
case err == nil && !panicked:
// A clean return before shutdown means the worker considers its
// job finished. Respect that rather than spinning it forever.
return
case panicked:
a.log.Error("background.panicked", "category", "worker",
"error_type", "worker_panic", "error_msg", err.Error(),
"restart_in_ms", backoff.Milliseconds())
default:
a.log.Error("background.failed", "category", "worker",
"error_type", "worker_error", "error_msg", err.Error(),
"restart_in_ms", backoff.Milliseconds())
}
select {
case <-ctx.Done():
return
case <-time.After(backoff):
}
if backoff *= 2; backoff > maxBackoff {
backoff = maxBackoff
}
}
}
// runBackgroundOnce invokes fn, converting a panic into an error so the
// supervisor can treat both failure modes alike. The stack is attached because
// a recovered panic with no stack is nearly unactionable.
func runBackgroundOnce(ctx context.Context, fn func(context.Context) error) (err error, panicked bool) {
defer func() {
if r := recover(); r != nil {
panicked = true
err = fmt.Errorf("panic: %v\n%s", r, debug.Stack())
}
}()
err = fn(ctx)
if errors.Is(err, context.Canceled) {
err = nil
}
return err, false
}

104
vendor/github.com/orchard9/go-chassis/chassis/auth.go generated vendored Normal file
View File

@ -0,0 +1,104 @@
package chassis
import (
"context"
"crypto/subtle"
"net/http"
"strings"
)
// Identity is the authenticated principal attached to the request context by an
// auth Middleware. Subject is the stable principal id; OrgID is the tenant the
// principal is acting in (the multi-tenant scoping key — every tenant row
// filters by it); Scopes are the granted permissions; Claims carries any extra
// verified attributes without coupling chassis to a domain type.
type Identity struct {
Subject string
OrgID string
Scopes []string
Claims map[string]string
}
// HasScope reports whether the identity holds the required scope. A held scope
// ending in ":*" dominates any required scope sharing its prefix, so "admin:*"
// satisfies "admin:read". An exact match always passes.
func (id *Identity) HasScope(required string) bool {
if id == nil {
return false
}
for _, s := range id.Scopes {
if s == required {
return true
}
if prefix, ok := strings.CutSuffix(s, ":*"); ok {
if reqPrefix, _, found := strings.Cut(required, ":"); found && reqPrefix == prefix {
return true
}
}
}
return false
}
// Authenticator verifies a request and returns its Identity, or an *Error
// (Unauthorized/Forbidden) when verification fails. Implementations:
// - NoAuth — local default, everyone is anonymous (documented seam).
// - StaticToken — shared bearer token, constant-time compared (local/CI).
// - (later) OIDC — coreos/go-oidc verifier, JWKS cache, refresh-on-kid-miss.
// - (later) APIKey — hashed key behind a Secrets port, constant-time compare.
type Authenticator interface {
Authenticate(r *http.Request) (*Identity, error)
}
type identityKey struct{}
// IdentityFrom returns the authenticated identity, or false when the route was
// not behind RequireAuth.
func IdentityFrom(ctx context.Context) (*Identity, bool) {
id, ok := ctx.Value(identityKey{}).(*Identity)
return id, ok
}
// RequireAuth is route/group middleware that runs the Authenticator, rejecting
// the request (deny-by-default) when it fails and otherwise stashing the
// Identity in the context for handlers.
func RequireAuth(a Authenticator) Middleware {
return func(next HandlerFunc) HandlerFunc {
return func(c *Context) error {
id, err := a.Authenticate(c.r)
if err != nil {
return err // already an *Error (Unauthorized/Forbidden)
}
c.r = c.r.WithContext(context.WithValue(c.r.Context(), identityKey{}, id))
return next(c)
}
}
}
// NoAuth treats every caller as anonymous. The explicit local default — never
// select it for a protected environment.
type NoAuth struct{}
func (NoAuth) Authenticate(*http.Request) (*Identity, error) {
return &Identity{Subject: "anonymous"}, nil
}
// StaticToken authenticates a single shared bearer token in constant time
// (crypto/subtle) to avoid leaking the token via response timing. For local/CI
// and internal service-to-service; real principals use OIDC/API-key later.
type StaticToken struct{ token string }
// NewStaticToken builds a StaticToken; an empty token rejects every request.
func NewStaticToken(token string) StaticToken { return StaticToken{token: token} }
func (s StaticToken) Authenticate(r *http.Request) (*Identity, error) {
const prefix = "Bearer "
h := r.Header.Get("Authorization")
if s.token == "" || !strings.HasPrefix(h, prefix) {
return nil, Unauthorized("missing or malformed bearer token")
}
got := strings.TrimPrefix(h, prefix)
if subtle.ConstantTimeCompare([]byte(got), []byte(s.token)) != 1 {
return nil, Unauthorized("invalid token")
}
return &Identity{Subject: "static-token"}, nil
}

37
vendor/github.com/orchard9/go-chassis/chassis/authz.go generated vendored Normal file
View File

@ -0,0 +1,37 @@
package chassis
// Authorization middleware — the layer above authentication. RequireAuth proves
// WHO; these prove WHAT they may do and WHICH tenant they act in. Apply after
// RequireAuth on a route group.
// RequireScope rejects callers lacking the given scope (scope dominance applies:
// a held "x:*" satisfies "x:read"). 401 if unauthenticated, 403 if under-scoped.
func RequireScope(scope string) Middleware {
return func(next HandlerFunc) HandlerFunc {
return func(c *Context) error {
id, ok := c.Identity()
if !ok {
return Unauthorized("authentication required")
}
if !id.HasScope(scope) {
return Forbidden("missing required scope: " + scope)
}
return next(c)
}
}
}
// RequireOrg enforces the multi-tenant invariant: the caller must have a resolved
// active organization. Handlers behind it can rely on a non-empty Context.OrgID
// and MUST filter every tenant query by it.
func RequireOrg() Middleware {
return func(next HandlerFunc) HandlerFunc {
return func(c *Context) error {
id, ok := c.Identity()
if !ok || id.OrgID == "" {
return Forbidden("no active organization")
}
return next(c)
}
}
}

View File

@ -0,0 +1,139 @@
package chassis
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"strconv"
"strings"
"github.com/orchard9/go-chassis/logging"
)
// HandlerFunc is the chassis handler signature: return an error and the
// framework maps it to the JSON error envelope. See patterns/go-chassis.md.
type HandlerFunc func(*Context) error
// Middleware decorates a HandlerFunc (route/group scope: auth, rate-limit).
// Cross-cutting infra (recover, request-id, logging, metrics) is applied once
// at the server edge as http.Handler middleware, not here.
type Middleware func(HandlerFunc) HandlerFunc
// Context carries the request/response for one call plus typed helpers.
type Context struct {
w http.ResponseWriter
r *http.Request
maxBytes int64
validate func(any) error // optional; set from Config.Validator
// closing fires when the app begins draining, so a Stream ends with the
// pod instead of holding Shutdown open.
closing <-chan struct{}
}
// Context returns the request context (deadline + request-scoped logger).
func (c *Context) Context() context.Context { return c.r.Context() }
// Request exposes the raw request for the rare case a helper does not cover.
func (c *Context) Request() *http.Request { return c.r }
// Writer exposes the raw ResponseWriter for the three responses the JSON
// envelope cannot carry: the Scalar docs page (HTML), the OpenAPI document
// (pre-rendered bytes), and a streamed artifact (video/mp4).
func (c *Context) Writer() http.ResponseWriter { return c.w }
// Log returns the request-scoped logger (carries request_id).
func (c *Context) Log() *slog.Logger { return logging.From(c.r.Context()) }
// PathValue returns a ServeMux wildcard value, e.g. {id} from "/v1/x/{id}".
func (c *Context) PathValue(key string) string { return c.r.PathValue(key) }
// Identity returns the authenticated principal, or false when the route was not
// behind RequireAuth.
func (c *Context) Identity() (*Identity, bool) { return IdentityFrom(c.r.Context()) }
// OrgID returns the active tenant for the caller ("" when unauthenticated or the
// identity carries no org). Repos MUST filter every tenant query by it.
func (c *Context) OrgID() string {
if id, ok := c.Identity(); ok {
return id.OrgID
}
return ""
}
// TraceID returns the request's trace id (W3C traceparent or generated).
func (c *Context) TraceID() string {
id, _ := TraceIDFrom(c.r.Context())
return id
}
// Bind enforces the body-size limit, then JSON-decodes into v rejecting unknown
// fields. An oversized body becomes 413; a malformed body becomes 400 — neither
// leaks internals to the client.
func (c *Context) Bind(v any) error {
c.r.Body = http.MaxBytesReader(c.w, c.r.Body, c.maxBytes)
dec := json.NewDecoder(c.r.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(v); err != nil {
var maxErr *http.MaxBytesError
if errors.As(err, &maxErr) {
return PayloadTooLarge("request body too large").WithCause(err)
}
if f, ok := unknownField(err); ok {
return BadRequest(fmt.Sprintf("unknown field %q", f)).WithCause(err)
}
return BadRequest("invalid JSON body").WithCause(err)
}
if c.validate != nil {
if err := c.validate(v); err != nil {
return Unprocessable("request failed validation").WithCause(err)
}
}
return nil
}
// unknownFieldPrefix is what encoding/json returns under
// DisallowUnknownFields. The stdlib gives no typed error for it, so matching the
// text is the only way to tell "you sent a field we do not accept" from "your
// JSON is broken" — and they are different bugs on the caller's side.
const unknownFieldPrefix = `json: unknown field `
// unknownField extracts the rejected field name, if that is why decoding failed.
//
// The name is echoed to the client because the alternative is what this cost us:
// a worker POSTing a well-formed body with one extra key is told "invalid JSON
// body", which is false, and has nothing to act on. Echoing a key the caller
// just sent leaks nothing. The rest of the decoder's errors stay generic — an
// UnmarshalTypeError names Go struct fields and types, which is internal detail.
func unknownField(err error) (string, bool) {
msg := err.Error()
if !strings.HasPrefix(msg, unknownFieldPrefix) {
return "", false
}
name, uerr := strconv.Unquote(strings.TrimPrefix(msg, unknownFieldPrefix))
if uerr != nil {
return "", false
}
return name, true
}
// OK writes 200 + JSON. Created writes 201. NoContent writes 204.
func (c *Context) OK(v any) error { return c.JSON(http.StatusOK, v) }
func (c *Context) Created(v any) error { return c.JSON(http.StatusCreated, v) }
func (c *Context) NoContent() error {
c.w.WriteHeader(http.StatusNoContent)
return nil
}
// JSON writes the status code and JSON-encodes v.
func (c *Context) JSON(code int, v any) error {
c.w.Header().Set("Content-Type", "application/json; charset=utf-8")
c.w.WriteHeader(code)
if v == nil {
return nil
}
return json.NewEncoder(c.w).Encode(v)
}

View File

@ -0,0 +1,74 @@
package chassis
import (
"errors"
"fmt"
"net/http"
)
// Error is the HTTP error model: a stable machine Code + a SAFE public Msg
// (no secrets, no internal detail) shown to the client, plus an internal cause
// that is logged server-side and NEVER serialized to the response. Handlers
// return these; the framework maps them to the JSON error envelope.
//
// This is distinct from any domain result/status enum a handler returns as a
// SUCCESS-response field — those are 200 bodies, not errors.
type Error struct {
Status int // HTTP status code
Code string // stable machine-readable code (snake_case)
Msg string // safe public message
cause error // internal cause: logged, never sent to the client
}
func (e *Error) Error() string {
if e.cause != nil {
return fmt.Sprintf("%s: %s: %v", e.Code, e.Msg, e.cause)
}
return fmt.Sprintf("%s: %s", e.Code, e.Msg)
}
func (e *Error) Unwrap() error { return e.cause }
// WithCause attaches the internal cause (logged, never sent). Returns a copy so
// the package-level constructors stay immutable.
func (e *Error) WithCause(err error) *Error {
c := *e
c.cause = err
return &c
}
func newErr(status int, code, msg string) *Error {
return &Error{Status: status, Code: code, Msg: msg}
}
// Constructors — public Msg defaults are safe and generic; override per call.
func BadRequest(msg string) *Error { return newErr(http.StatusBadRequest, "bad_request", msg) }
func Unauthorized(msg string) *Error { return newErr(http.StatusUnauthorized, "unauthorized", msg) }
func Forbidden(msg string) *Error { return newErr(http.StatusForbidden, "forbidden", msg) }
func NotFound(msg string) *Error { return newErr(http.StatusNotFound, "not_found", msg) }
func Conflict(msg string) *Error { return newErr(http.StatusConflict, "conflict", msg) }
func Unprocessable(msg string) *Error {
return newErr(http.StatusUnprocessableEntity, "unprocessable", msg)
}
func TooManyRequests(msg string) *Error {
return newErr(http.StatusTooManyRequests, "rate_limited", msg)
}
func PayloadTooLarge(msg string) *Error {
return newErr(http.StatusRequestEntityTooLarge, "payload_too_large", msg)
}
// Internal wraps any non-Error as a 500 with a generic public message; the real
// cause is preserved for logging but hidden from the client.
func Internal(cause error) *Error {
return (&Error{Status: http.StatusInternalServerError, Code: "internal", Msg: "internal error"}).WithCause(cause)
}
// asError maps any error to an *Error, defaulting unknown errors to a 500 whose
// detail is hidden from the client.
func asError(err error) *Error {
var e *Error
if errors.As(err, &e) {
return e
}
return Internal(err)
}

View File

@ -0,0 +1,50 @@
package chassis
import (
"context"
"net/http"
)
// CheckFunc reports whether a dependency is usable right now. It MUST honor the
// passed ctx deadline and return a non-nil error (with no secrets) when unhealthy.
type CheckFunc func(ctx context.Context) error
type namedCheck struct {
name string
fn CheckFunc
}
// handleLive is the k8s liveness probe: 200 while the process runs. It does NOT
// check dependencies and stays 200 during drain — a flapping dep or an
// in-progress shutdown must never trigger a restart, only LB removal.
func (a *App) handleLive(c *Context) error {
return c.OK(map[string]any{"status": "ok"})
}
// handleReady is the k8s readiness probe. It returns 503 the moment shutdown
// begins (two-phase drain: the LB pulls the pod before connections close), and
// otherwise 200 only when every registered dependency check passes.
func (a *App) handleReady(c *Context) error {
if a.draining.Load() {
return c.JSON(http.StatusServiceUnavailable, map[string]any{"status": "draining"})
}
ctx, cancel := context.WithTimeout(c.Context(), a.cfg.ReadyTimeout)
defer cancel()
checks := make(map[string]string, len(a.checks))
ok := true
for _, ch := range a.checks {
if err := ch.fn(ctx); err != nil {
checks[ch.name] = "error: " + err.Error()
ok = false
continue
}
checks[ch.name] = "ok"
}
status, code := "ok", http.StatusOK
if !ok {
status, code = "unready", http.StatusServiceUnavailable
}
return c.JSON(code, map[string]any{"status": status, "checks": checks})
}

View File

@ -0,0 +1,119 @@
package chassis
import (
"bytes"
"context"
"net/http"
"time"
)
// StoredResponse is a cached successful response, replayed verbatim (status +
// headers + body) on a duplicate Idempotency-Key.
type StoredResponse struct {
Status int
Header http.Header
Body []byte
}
// IdempotencyStore persists and replays responses keyed by Idempotency-Key.
// Back it with Redis (TTL'd) in shared/adapters/redis. Lookup returns found=false
// when the key is new; Save records a successful response.
type IdempotencyStore interface {
Lookup(ctx context.Context, key string) (StoredResponse, bool, error)
Save(ctx context.Context, key string, resp StoredResponse, ttl time.Duration) error
}
// Idempotent dedups mutating requests by the Idempotency-Key header: a duplicate
// returns the original response without re-running the handler. Apply it to
// POST/PUT route groups. Requests without the header pass straight through. The
// key is scoped by the caller's org so keys can't collide across tenants. The
// store is best-effort: a backend error fails open (the request proceeds) rather
// than blocking writes.
func Idempotent(store IdempotencyStore, ttl time.Duration) Middleware {
return func(next HandlerFunc) HandlerFunc {
return func(c *Context) error {
key := c.r.Header.Get("Idempotency-Key")
if key == "" {
return next(c)
}
// Scope by tenant. With no resolved org we refuse to cache rather than
// share a global (cross-tenant) key — pair this with RequireOrg.
org := c.OrgID()
if org == "" {
c.Log().Warn("idempotency.skipped_no_org", "category", "idempotency")
return next(c)
}
scoped := org + ":" + key
ctx := c.r.Context()
if sr, found, err := store.Lookup(ctx, scoped); err != nil {
c.Log().Warn("idempotency.lookup_failed", "category", "idempotency", "error_msg", err.Error())
} else if found {
h := c.w.Header()
for k, vs := range sr.Header { // replay the original headers verbatim
for _, v := range vs {
h.Add(k, v)
}
}
h.Set("Idempotency-Replayed", "true")
c.w.WriteHeader(sr.Status)
_, werr := c.w.Write(sr.Body)
return werr
}
// Capture the handler's response so we can persist + flush it. Restore
// the writer via defer so a panic in next() can't leave c.w dangling.
rec := &captureWriter{header: http.Header{}, status: http.StatusOK}
orig := c.w
c.w = rec
defer func() { c.w = orig }()
if err := next(c); err != nil {
return err // errors aren't cached; the framework writes the envelope to orig
}
for k, vs := range rec.header {
for _, v := range vs {
orig.Header().Add(k, v)
}
}
orig.WriteHeader(rec.status)
if _, werr := orig.Write(rec.buf.Bytes()); werr != nil {
return werr
}
if rec.status >= 200 && rec.status < 300 {
body := append([]byte(nil), rec.buf.Bytes()...)
stored := StoredResponse{Status: rec.status, Header: rec.header.Clone(), Body: body}
if serr := store.Save(ctx, scoped, stored, ttl); serr != nil {
c.Log().Warn("idempotency.save_failed", "category", "idempotency", "error_msg", serr.Error())
}
}
return nil
}
}
}
// captureWriter buffers a handler's response so the idempotency middleware can
// persist and replay it. It implements http.ResponseWriter.
type captureWriter struct {
header http.Header
buf bytes.Buffer
status int
wrote bool
}
func (c *captureWriter) Header() http.Header { return c.header }
func (c *captureWriter) WriteHeader(code int) {
if c.wrote {
return
}
c.status = code
c.wrote = true
}
func (c *captureWriter) Write(b []byte) (int, error) {
if !c.wrote {
c.WriteHeader(http.StatusOK)
}
return c.buf.Write(b)
}

View File

@ -0,0 +1,67 @@
package chassis
import (
"net/http"
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/collectors"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
// metrics holds the RED HTTP collectors on a private registry (no global state,
// so tests and multiple apps never collide). The route label is the matched
// ServeMux pattern — bounded cardinality, never the raw path.
type metrics struct {
reg *prometheus.Registry
reqs *prometheus.CounterVec
dur *prometheus.HistogramVec
inflight prometheus.Gauge
}
// newMetrics builds the RED collectors and registers them alongside any
// service-owned collectors (Config.Collectors) on the same private registry —
// so /metrics is one scrape and a domain gauge cannot be lost to a second
// endpoint nobody remembers to scrape.
func newMetrics(service string, extra ...prometheus.Collector) *metrics {
reg := prometheus.NewRegistry()
labels := prometheus.Labels{"service": service}
m := &metrics{
reg: reg,
reqs: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total", Help: "Total HTTP requests.", ConstLabels: labels,
}, []string{"method", "route", "status"}),
dur: prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds", Help: "HTTP request latency.",
Buckets: prometheus.DefBuckets, ConstLabels: labels,
}, []string{"method", "route"}),
inflight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "http_requests_in_flight", Help: "In-flight HTTP requests.", ConstLabels: labels,
}),
}
reg.MustRegister(m.reqs, m.dur, m.inflight,
collectors.NewGoCollector(),
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))
for _, c := range extra {
reg.MustRegister(c)
}
return m
}
func (m *metrics) observe(method, route string, status int, d time.Duration) {
m.countOnly(method, route, status)
m.dur.WithLabelValues(method, route).Observe(d.Seconds())
}
// countOnly records a request without timing it. For responses whose elapsed
// time is not a latency — an event stream ends when the operator closes the
// tab, and putting that in the histogram makes every latency alert lie.
func (m *metrics) countOnly(method, route string, status int) {
m.reqs.WithLabelValues(method, route, strconv.Itoa(status)).Inc()
}
// handler serves the Prometheus exposition for the metrics agent to scrape.
func (m *metrics) handler() http.Handler {
return promhttp.HandlerFor(m.reg, promhttp.HandlerOpts{Registry: m.reg})
}

View File

@ -0,0 +1,244 @@
package chassis
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"net/http"
"runtime/debug"
"strconv"
"strings"
"time"
"github.com/orchard9/go-chassis/logging"
)
// chain composes edge middleware so mw[0] is outermost (runs first).
func chain(h http.Handler, mw ...func(http.Handler) http.Handler) http.Handler {
for i := len(mw) - 1; i >= 0; i-- {
h = mw[i](h)
}
return h
}
// statusRecorder captures the response status for logging/metrics and survives
// double WriteHeader; it forwards Flush so SSE/streaming handlers still work.
type statusRecorder struct {
http.ResponseWriter
status int
wrote bool
// streamed is set by Context.Stream. It keeps a connection an operator
// held open for twenty minutes out of the request-latency histogram.
streamed bool
}
func (r *statusRecorder) WriteHeader(code int) {
if r.wrote {
return
}
r.status = code
r.wrote = true
r.ResponseWriter.WriteHeader(code)
}
func (r *statusRecorder) Write(b []byte) (int, error) {
if !r.wrote {
r.WriteHeader(http.StatusOK)
}
return r.ResponseWriter.Write(b)
}
func (r *statusRecorder) Flush() {
if f, ok := r.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// Unwrap exposes the wrapped writer to http.ResponseController, which is how a
// streaming handler clears the server's WriteTimeout for its own connection.
// Without it ResponseController stops at this wrapper and every deadline call
// returns ErrNotSupported — the response is then cut mid-stream at
// RequestTimeout+socketHeadroom with no error the handler can report.
func (r *statusRecorder) Unwrap() http.ResponseWriter { return r.ResponseWriter }
func newRequestID() string {
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
return strconv.FormatInt(time.Now().UnixNano(), 16)
}
return hex.EncodeToString(b[:])
}
type traceIDKey struct{}
// TraceIDFrom returns the request's trace id, or false outside a chassis-handled
// request. shared/httpclient uses it to propagate traceparent downstream.
func TraceIDFrom(ctx context.Context) (string, bool) {
id, ok := ctx.Value(traceIDKey{}).(string)
return id, ok
}
// traceIDFromHeader extracts the 32-hex trace-id from a W3C traceparent header
// ("00-<32hex>-<16hex>-<flags>"); on absence/malformation it generates one.
func traceIDFromHeader(traceparent string) string {
parts := strings.Split(traceparent, "-")
if len(parts) == 4 && len(parts[1]) == 32 && parts[1] != strings.Repeat("0", 32) {
if _, err := hex.DecodeString(parts[1]); err == nil {
return parts[1]
}
}
return newRequestID()
}
// instrument is the core edge middleware: it assigns/propagates a request_id,
// builds the request-scoped logger, applies the per-request timeout, counts
// in-flight requests, RECOVERS panics (so the access log + metrics observe the
// real 500), and emits one access line + RED metrics. The route label is the
// matched ServeMux pattern (bounded cardinality), never the raw path.
func (a *App) instrument(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id := r.Header.Get("X-Request-Id")
if id == "" {
id = newRequestID()
}
w.Header().Set("X-Request-Id", id)
// Derive the trace id from W3C traceparent (or generate one) so logs +
// downstream calls correlate. trace_id is a regular log field, NEVER a
// stream field (high cardinality).
trace := traceIDFromHeader(r.Header.Get("traceparent"))
w.Header().Set("X-Trace-Id", trace)
log := a.log.With("request_id", id, "trace_id", trace)
ctx := context.WithValue(r.Context(), traceIDKey{}, trace)
ctx = logging.Into(ctx, log)
if a.cfg.RequestTimeout > 0 {
// A deadline cannot be removed from a derived context, so the
// pre-timeout one is carried alongside for Context.Stream: a
// server-sent-events response is open for minutes by design and
// must stay bound to client disconnect and drain, not to the
// per-request budget every other route wants.
ctx = context.WithValue(ctx, untimedKey{}, ctx)
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, a.cfg.RequestTimeout)
defer cancel()
}
r2 := r.WithContext(ctx)
rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}
a.metrics.inflight.Inc()
defer a.metrics.inflight.Dec()
start := time.Now()
func() {
defer func() {
if rv := recover(); rv != nil {
log.Error("panic.recovered", "category", "panic",
"error_type", "panic", "error_msg", fmt.Sprint(rv),
"stack", string(debug.Stack()))
if !rec.wrote {
rec.Header().Set("Content-Type", "application/json; charset=utf-8")
rec.WriteHeader(http.StatusInternalServerError)
// Same envelope as writeError, request_id included. A panic
// is precisely when a user needs an id to quote, so this is
// the worst response to leave it out of.
_, _ = fmt.Fprintf(rec,
`{"error":{"code":"internal","message":"internal error","request_id":%q}}`, id)
}
}
}()
next.ServeHTTP(rec, r2)
}()
route := routeLabel(r2.Pattern)
dur := time.Since(start)
log.Debug("http.request",
"http_method", r.Method, "http_route", route, "http_path", r.URL.Path,
"http_status", rec.status, "http_duration_ms", dur.Milliseconds(),
"http_streamed", rec.streamed)
// A stream's elapsed time is how long an operator left a tab open, not
// how long the service took to answer. Feeding it to the latency
// histogram fired HighRequestLatency the first time somebody opened the
// console and would poison every latency panel for the whole service,
// so a streamed response is counted but not timed.
if rec.streamed {
a.metrics.countOnly(r.Method, route, rec.status)
return
}
a.metrics.observe(r.Method, route, rec.status, dur)
})
}
// routeLabel reduces a matched ServeMux pattern ("GET /v1/x/{id}") to its path
// shape ("/v1/x/{id}") for a bounded metrics/log label — never the raw path.
func routeLabel(pattern string) string {
if pattern == "" {
return "other"
}
if i := strings.IndexByte(pattern, ' '); i >= 0 { // drop the "METHOD " prefix
pattern = pattern[i+1:]
}
return pattern
}
// secureHeaders sets conservative response headers for a JSON API (no inline
// content, no framing). HSTS is emitted only in prod, where TLS terminates.
func (a *App) secureHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
h.Set("X-Content-Type-Options", "nosniff")
h.Set("X-Frame-Options", "DENY")
h.Set("Referrer-Policy", "no-referrer")
h.Set("Content-Security-Policy", "default-src 'none'; frame-ancestors 'none'")
if a.cfg.Env == "prod" {
h.Set("Strict-Transport-Security", "max-age=31536000; includeSubDomains")
}
next.ServeHTTP(w, r)
})
}
// cors echoes the request Origin when it is on the allowlist and answers
// preflight. A platform serves more than one browser surface (admin, tenant,
// operator), so the allowlist is a set and the response echoes the matched
// origin rather than a single fixed value — echoing a fixed origin breaks every
// UI but one, and echoing the request unchecked is an open CORS policy.
func (a *App) cors(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
// The response body depends on Origin whenever an allowlist is
// configured, so Vary is set even on a non-match — otherwise a shared
// cache can serve an allowed origin's response to a denied one.
if len(a.cfg.AllowOrigins) > 0 {
h.Add("Vary", "Origin")
}
if origin := r.Header.Get("Origin"); origin != "" && a.originAllowed(origin) {
h.Set("Access-Control-Allow-Origin", origin)
h.Set("Access-Control-Allow-Methods", "GET, POST, PUT, PATCH, DELETE, OPTIONS")
h.Set("Access-Control-Allow-Headers", "Content-Type, Authorization, X-Request-Id, Idempotency-Key")
// Without this the browser can send X-Request-Id but cannot READ the
// one we echo back, so a cross-origin SPA has no id to show the user
// next to an error toast. Allow-Headers governs the request; only
// Expose-Headers governs what JS may read off the response.
h.Set("Access-Control-Expose-Headers", "X-Request-Id, X-Trace-Id")
h.Set("Access-Control-Max-Age", "600")
}
if r.Method == http.MethodOptions {
w.WriteHeader(http.StatusNoContent)
return
}
next.ServeHTTP(w, r)
})
}
// originAllowed reports whether origin is on the configured allowlist. Matching
// is exact: no suffix or wildcard matching, because "endswith example.com"
// also matches "evil-example.com".
func (a *App) originAllowed(origin string) bool {
for _, allowed := range a.cfg.AllowOrigins {
if allowed == origin {
return true
}
}
return false
}

View File

@ -0,0 +1,44 @@
package chassis
import "strconv"
// Page is a parsed cursor-pagination request (?limit=&cursor=). Cursor is opaque
// to the chassis; handlers encode/decode it (e.g. an id or keyset token).
type Page struct {
Limit int
Cursor string
}
// Page parses pagination params from the query string, clamping limit to
// [1, maxLimit] and defaulting to defLimit.
func (c *Context) Page(defLimit, maxLimit int) Page {
q := c.r.URL.Query()
p := Page{Limit: defLimit, Cursor: q.Get("cursor")}
if v := q.Get("limit"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
p.Limit = n
}
}
if p.Limit > maxLimit {
p.Limit = maxLimit
}
if p.Limit < 1 {
p.Limit = 1
}
return p
}
// ListResponse is the standard list envelope. NextCursor is "" on the last page.
type ListResponse[T any] struct {
Items []T `json:"items"`
NextCursor string `json:"next_cursor,omitempty"`
}
// List builds a ListResponse, normalizing a nil slice to []. nextCursor is the
// token a client passes as ?cursor= to fetch the next page ("" = no more).
func List[T any](items []T, nextCursor string) ListResponse[T] {
if items == nil {
items = []T{}
}
return ListResponse[T]{Items: items, NextCursor: nextCursor}
}

View File

@ -0,0 +1,89 @@
package chassis
import (
"context"
"math"
"net"
"net/http"
"strconv"
"strings"
"time"
)
// RateLimiter decides whether a key may proceed now. Back it with a Redis token
// bucket in shared/adapters/redis (tiered by actor). retryAfter hints when to
// retry; it is surfaced as the Retry-After header on a 429.
type RateLimiter interface {
Allow(ctx context.Context, key string) (allowed bool, retryAfter time.Duration, err error)
}
// RateLimit rejects requests over the limit with 429 + Retry-After. keyFn maps a
// request to a bucket key (default: authenticated subject, else client IP). The
// limiter is best-effort: a backend error fails open (request proceeds) so a
// degraded Redis can't take down the API.
func RateLimit(limiter RateLimiter, keyFn func(*Context) string) Middleware {
if keyFn == nil {
keyFn = DefaultRateKey
}
return func(next HandlerFunc) HandlerFunc {
return func(c *Context) error {
allowed, retryAfter, err := limiter.Allow(c.r.Context(), keyFn(c))
if err != nil {
c.Log().Warn("ratelimit.failed", "category", "ratelimit", "error_msg", err.Error())
return next(c)
}
if !allowed {
if retryAfter > 0 {
c.w.Header().Set("Retry-After", strconv.Itoa(int(math.Ceil(retryAfter.Seconds()))))
}
return TooManyRequests("rate limit exceeded")
}
return next(c)
}
}
}
// DefaultRateKey buckets by authenticated subject when present, else by client
// IP. It does NOT trust X-Forwarded-For (trustedHops=0), so a spoofed XFF can't
// mint a fresh bucket per request. Behind N trusted proxies/LBs, use RateKey(N).
func DefaultRateKey(c *Context) string { return rateKey(c, 0) }
// RateKey returns a key function for a deployment behind trustedHops proxies/LBs
// (the IP is taken trustedHops entries from the right of X-Forwarded-For).
func RateKey(trustedHops int) func(*Context) string {
return func(c *Context) string { return rateKey(c, trustedHops) }
}
func rateKey(c *Context, trustedHops int) string {
if id, ok := c.Identity(); ok && id.Subject != "" && id.Subject != "anonymous" {
return "sub:" + id.Subject
}
return "ip:" + clientIP(c.r, trustedHops)
}
// clientIP derives the client address. With trustedHops<=0 it uses the connection
// RemoteAddr (XFF is attacker-controlled and ignored). Behind trustedHops trusted
// proxies it takes the entry that many hops from the right of X-Forwarded-For —
// the first address the trust boundary did not append.
func clientIP(r *http.Request, trustedHops int) string {
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
host = r.RemoteAddr
}
if trustedHops <= 0 {
return host
}
xff := r.Header.Get("X-Forwarded-For")
if xff == "" {
return host
}
parts := strings.Split(xff, ",")
idx := len(parts) - trustedHops
if idx < 0 || idx >= len(parts) {
return host
}
if v := strings.TrimSpace(parts[idx]); v != "" {
return v
}
return host
}

154
vendor/github.com/orchard9/go-chassis/chassis/stream.go generated vendored Normal file
View File

@ -0,0 +1,154 @@
package chassis
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
// untimedKey carries the pre-deadline request context so Stream can shed the
// per-request budget. See instrument.
type untimedKey struct{}
// heartbeatEvery bounds how long a stream stays silent. Any proxy between the
// board and a browser will drop an idle connection eventually — Traefik's
// default is 0 (never) but Cloudflare's is 100s and a corporate egress proxy's
// is anyone's guess — so a comment frame goes out on every tick even when the
// board state has not moved. It is two bytes and it is the difference between
// a live dashboard and one that silently stopped updating an hour ago.
const heartbeatEvery = 20 * time.Second
// Stream is an open server-sent-events response.
//
// SSE rather than a websocket because the traffic is strictly one-way (the
// board pushes state, the browser never talks back), it survives every proxy
// that speaks HTTP/1.1, and EventSource reconnects on its own — so a board
// restart costs the dashboard a few seconds rather than a page reload.
type Stream struct {
w http.ResponseWriter
rc *http.ResponseController
ctx context.Context
closing <-chan struct{}
}
// Stream converts the response into an event stream and returns a handle.
//
// It clears this connection's write deadline (the server sets one from
// RequestTimeout for every other route) and detaches from the per-request
// context deadline, leaving the stream bound to exactly two things: the client
// hanging up, and the server beginning to drain.
//
// The handler MUST NOT write to the Context afterwards — the response is
// committed the moment this returns.
func (c *Context) Stream() (*Stream, error) {
w := c.w
rc := http.NewResponseController(w)
// A stream lives past any per-request deadline by definition. This is the
// call that needs statusRecorder.Unwrap; without it the connection is cut
// mid-stream at RequestTimeout+socketHeadroom.
if err := rc.SetWriteDeadline(time.Time{}); err != nil {
return nil, fmt.Errorf("chassis: stream needs a deadline-capable writer: %w", err)
}
// Tell the edge not to time this response. How long a stream stays open is
// how long an operator left a tab open; in the latency histogram it fires
// HighRequestLatency and drags every percentile for the whole service.
if rec, ok := w.(*statusRecorder); ok {
rec.streamed = true
}
ctx := c.r.Context()
if untimed, ok := ctx.Value(untimedKey{}).(context.Context); ok {
ctx = untimed
}
h := w.Header()
h.Set("Content-Type", "text/event-stream")
h.Set("Cache-Control", "no-cache, no-transform")
h.Set("Connection", "keep-alive")
// Traefik does not buffer, but this response passes through whatever the
// operator puts in front of it and an accumulating proxy turns a live
// stream into a batch delivered at close.
h.Set("X-Accel-Buffering", "no")
w.WriteHeader(http.StatusOK)
// Past this point the response is committed, so a flush failure must not
// become a returned error — the chassis would write a JSON envelope on top
// of a 200 event stream. The first Send surfaces a dead connection.
_ = rc.Flush()
return &Stream{w: w, rc: rc, ctx: ctx, closing: c.closing}, nil
}
// Context is the stream's lifetime: cancelled when the client disconnects.
func (s *Stream) Context() context.Context { return s.ctx }
// Send JSON-encodes v as one named event and flushes it. A write error means
// the client is gone; the caller returns and the handler ends.
func (s *Stream) Send(event string, v any) error {
body, err := json.Marshal(v)
if err != nil {
return fmt.Errorf("chassis: stream encode %s: %w", event, err)
}
// The payload is compact JSON from encoding/json, so it contains no raw
// newline and needs no multi-line data: continuation.
if _, err := fmt.Fprintf(s.w, "event: %s\ndata: %s\n\n", event, body); err != nil {
return err
}
return s.rc.Flush()
}
// Run pushes an initial frame, then one per tick, until the client disconnects
// or the server drains. It returns nil on every ordinary end — a browser
// closing a tab is not a server error and must not be logged as one.
func (s *Stream) Run(every time.Duration, frame func(context.Context) (any, error)) error {
send := func() error {
v, err := frame(s.ctx)
if err != nil {
return err
}
return s.Send("state", v)
}
if err := send(); err != nil {
return s.classify(err)
}
tick := time.NewTicker(every)
defer tick.Stop()
beat := time.NewTicker(heartbeatEvery)
defer beat.Stop()
for {
select {
case <-s.ctx.Done():
return nil
case <-s.closing:
// Tell the browser to come back rather than letting it infer a
// dead board from a closed socket.
_ = s.Send("bye", map[string]string{"reason": "draining"})
return nil
case <-beat.C:
if _, err := fmt.Fprint(s.w, ": ping\n\n"); err != nil {
return nil
}
if err := s.rc.Flush(); err != nil {
return nil
}
case <-tick.C:
if err := send(); err != nil {
return s.classify(err)
}
}
}
}
// classify swallows the errors that mean "the client left". A disconnect races
// every write, so treating it as a failure would fill the log with 500s every
// time somebody closes a dashboard tab.
func (s *Stream) classify(err error) error {
if s.ctx.Err() != nil {
return nil
}
return err
}

147
vendor/github.com/orchard9/go-chassis/config/config.go generated vendored Normal file
View File

@ -0,0 +1,147 @@
// Package config is the typed, validated environment loader used at the
// composition root (services/*/main.go, workers/*/main.go) — the ONLY place env
// is read. A Loader accumulates parse/validation errors so boot fails closed
// with one actionable message instead of silently running on bad config.
//
// Secrets (DB passwords, API keys) do NOT belong here — they come from a
// secret source (manager / file). This loads non-secret wiring only.
package config
import (
"errors"
"fmt"
"os"
"strconv"
"strings"
"time"
)
// Loader reads env vars with defaults and validation, collecting every error.
type Loader struct {
errs []error
}
// New returns an empty Loader.
func New() *Loader { return &Loader{} }
// String returns the env value or def when unset/empty.
func (l *Loader) String(key, def string) string {
if v := strings.TrimSpace(os.Getenv(key)); v != "" {
return v
}
return def
}
// Strings splits a comma-separated env value, trimming each entry and dropping
// empties, or returns def when unset. Used for allowlists (CORS origins,
// trusted proxies) where a platform has more than one legitimate value and a
// single-string field would silently serve only the first.
func (l *Loader) Strings(key string, def []string) []string {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
parts := strings.Split(raw, ",")
out := make([]string, 0, len(parts))
for _, p := range parts {
if v := strings.TrimSpace(p); v != "" {
out = append(out, v)
}
}
if len(out) == 0 {
return def
}
return out
}
// Required returns the env value or records an error when unset/empty.
func (l *Loader) Required(key string) string {
v := strings.TrimSpace(os.Getenv(key))
if v == "" {
l.errs = append(l.errs, fmt.Errorf("%s is required", key))
}
return v
}
// Int parses an integer env value, recording an error on a malformed value.
func (l *Loader) Int(key string, def int) int {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
n, err := strconv.Atoi(raw)
if err != nil {
l.errs = append(l.errs, fmt.Errorf("%s: invalid int %q", key, raw))
return def
}
return n
}
// Int64 parses a 64-bit integer env value, recording an error on a malformed
// value. Separate from Int because byte-size limits (body caps, artifact
// ceilings) legitimately exceed a 32-bit int on some platforms.
func (l *Loader) Int64(key string, def int64) int64 {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
n, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
l.errs = append(l.errs, fmt.Errorf("%s: invalid int64 %q", key, raw))
return def
}
return n
}
// Port parses and bounds-checks a TCP port (1..65535).
func (l *Loader) Port(key string, def int) int {
n := l.Int(key, def)
if n < 1 || n > 65535 {
l.errs = append(l.errs, fmt.Errorf("%s: port %d out of range 1..65535", key, n))
return def
}
return n
}
// Duration parses a Go duration (e.g. 5s, 1h), recording an error if malformed.
func (l *Loader) Duration(key string, def time.Duration) time.Duration {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
d, err := time.ParseDuration(raw)
if err != nil {
l.errs = append(l.errs, fmt.Errorf("%s: invalid duration %q", key, raw))
return def
}
return d
}
// Bool parses a boolean (1/t/true/0/f/false), recording an error if malformed.
func (l *Loader) Bool(key string, def bool) bool {
raw := strings.TrimSpace(os.Getenv(key))
if raw == "" {
return def
}
b, err := strconv.ParseBool(raw)
if err != nil {
l.errs = append(l.errs, fmt.Errorf("%s: invalid bool %q", key, raw))
return def
}
return b
}
// OneOf returns the env value when it is in allowed, else records an error.
func (l *Loader) OneOf(key, def string, allowed ...string) string {
v := l.String(key, def)
for _, a := range allowed {
if v == a {
return v
}
}
l.errs = append(l.errs, fmt.Errorf("%s: %q not one of %s", key, v, strings.Join(allowed, "|")))
return def
}
// Err returns the joined validation errors, or nil when the config is clean.
func (l *Loader) Err() error { return errors.Join(l.errs...) }

View File

@ -0,0 +1,25 @@
package logging
import (
"context"
"log/slog"
)
type ctxKey int
const loggerKey ctxKey = iota
// Into returns a child context carrying log, so request-scoped fields
// (request_id, route, ...) added at the edge flow to every downstream call.
func Into(ctx context.Context, log *slog.Logger) context.Context {
return context.WithValue(ctx, loggerKey, log)
}
// From returns the request-scoped logger stored in ctx, or the package fallback
// (set by SetFallback at boot) so a missing logger never panics or drops logs.
func From(ctx context.Context) *slog.Logger {
if log, ok := ctx.Value(loggerKey).(*slog.Logger); ok && log != nil {
return log
}
return fallback()
}

View File

@ -0,0 +1,139 @@
// Package logging is the core structured logger: one JSON object per line on
// stdout, ready for any log agent (Fluent Bit, vector, the cloud's native
// collector) to ship. Fields: ts (ISO-8601 UTC, ms, Z), level, service + env
// (the only indexed stream fields — keep them closed enums), msg, plus caller
// attrs. Secrets/PII MUST NOT be logged; forbiddenKeys redacts common offenders
// as defense-in-depth behind code review + lint. See patterns/go-chassis.md.
package logging
import (
"context"
"io"
"log/slog"
"os"
"strings"
"sync"
)
// LevelCritical sits above slog.LevelError; it maps to the "critical" enum and
// is reserved for fail-closed boot/serve refusal (wire it to a page).
const LevelCritical = slog.Level(12)
// Config selects the closed-enum stream fields the log store indexes.
type Config struct {
Service string // closed enum, keep small (e.g. api, job-worker)
Env string // dev | staging | prod
}
// New returns the core logger writing the JSON wire format to stdout.
func New(cfg Config) *slog.Logger { return newTo(os.Stdout, cfg) }
// NewTo returns the core logger writing the JSON wire format to w. Server mode
// uses New (stdout); a bounded maintenance command uses this to send its
// operational diagnostics to stderr, keeping stdout a clean machine-readable
// result stream a caller can parse without stripping log lines out of it.
func NewTo(w io.Writer, cfg Config) *slog.Logger { return newTo(w, cfg) }
func newTo(w io.Writer, cfg Config) *slog.Logger {
h := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: slog.LevelDebug,
ReplaceAttr: replace,
})
return slog.New(h).With("service", cfg.Service, "env", cfg.Env)
}
func replace(groups []string, a slog.Attr) slog.Attr {
if forbiddenKeys[a.Key] { // defense-in-depth secret/PII redaction
return slog.String(a.Key, "[REDACTED]")
}
// The built-in rewrites below apply only to the record's own time/level
// attrs, which slog always passes at the top level. A caller attr that
// happens to be named "time" or "level" — log.Info("m", "level", "high") —
// arrives here too, so both the group depth and the value kind are checked
// before converting. An unchecked assertion here panics the process inside
// the logger every binary depends on.
if len(groups) > 0 {
return a
}
switch a.Key {
case slog.TimeKey:
if a.Value.Kind() != slog.KindTime {
return a
}
// ts: ISO-8601 UTC, ms precision, Z suffix.
a.Key = "ts"
a.Value = slog.StringValue(a.Value.Time().UTC().Format("2006-01-02T15:04:05.000Z07:00"))
case slog.LevelKey:
lvl, ok := a.Value.Any().(slog.Level)
if !ok {
return a
}
a.Key = "level"
a.Value = slog.StringValue(levelString(lvl))
}
return a
}
func levelString(l slog.Level) string {
switch {
case l >= LevelCritical:
return "critical"
case l >= slog.LevelError:
return "error"
case l >= slog.LevelWarn:
return "warn"
case l >= slog.LevelInfo:
return "info"
default:
return "debug"
}
}
// forbiddenKeys are field names that MUST NOT reach the log corpus. Extend it
// with YOUR product's sensitive fields (PII, PII, financial, tokens).
var forbiddenKeys = map[string]bool{
"password": true, "passwd": true, "secret": true, "token": true,
"api_key": true, "apikey": true, "authorization": true, "cookie": true,
"ssn": true, "email": true, "phone": true,
}
var (
fallbackMu sync.RWMutex
fallbackLog *slog.Logger
)
// SetFallback installs the logger From returns when no request-scoped logger is
// in context. Set once at boot from the composition root.
func SetFallback(l *slog.Logger) {
fallbackMu.Lock()
defer fallbackMu.Unlock()
fallbackLog = l
}
func fallback() *slog.Logger {
fallbackMu.RLock()
l := fallbackLog
fallbackMu.RUnlock()
if l != nil {
return l
}
return New(Config{Service: "unknown", Env: "dev"})
}
// Critical logs at the fail-closed level (boot/serve refusal). Bind to a page.
func Critical(ctx context.Context, l *slog.Logger, msg string, args ...any) {
l.Log(ctx, LevelCritical, msg, args...)
}
// Env normalizes an APP_ENV value (local|dev|staging|prod, or a gcp-* / aws-*
// prefix) to the log env enum.
func Env(appEnv string) string {
switch {
case strings.HasSuffix(appEnv, "prod"):
return "prod"
case strings.HasSuffix(appEnv, "staging"):
return "staging"
default:
return "dev"
}
}

View File

@ -0,0 +1,53 @@
package logging
import "encoding/json"
// Forbidden reports whether key is a secrets/PII field name that must never be
// persisted in cleartext (logs, audit snapshots, alert payloads).
func Forbidden(key string) bool { return forbiddenKeys[key] }
// Redact returns "[REDACTED]" when key is forbidden, otherwise val. Use it when
// projecting attributes into a non-log sink (e.g. alert events).
func Redact(key, val string) string {
if forbiddenKeys[key] {
return "[REDACTED]"
}
return val
}
// RedactJSON returns a copy of a JSON document with every forbidden key's value
// (at any depth) replaced by "[REDACTED]". This is the defense-in-depth scrubber
// the audit sink applies to before/after snapshots so PII cannot land in the
// immutable audit log. Non-JSON or unparseable input is returned unchanged.
func RedactJSON(b []byte) []byte {
if len(b) == 0 {
return b
}
var v any
if err := json.Unmarshal(b, &v); err != nil {
return b
}
redactValue(v)
out, err := json.Marshal(v)
if err != nil {
return b
}
return out
}
func redactValue(v any) {
switch t := v.(type) {
case map[string]any:
for k, val := range t {
if forbiddenKeys[k] {
t[k] = "[REDACTED]"
continue
}
redactValue(val)
}
case []any:
for _, e := range t {
redactValue(e)
}
}
}

201
vendor/github.com/prometheus/client_golang/LICENSE generated vendored Normal file
View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

18
vendor/github.com/prometheus/client_golang/NOTICE generated vendored Normal file
View File

@ -0,0 +1,18 @@
Prometheus instrumentation library for Go applications
Copyright 2012-2015 The Prometheus Authors
This product includes software developed at
SoundCloud Ltd. (http://soundcloud.com/).
The following components are included in this product:
perks - a fork of https://github.com/bmizerany/perks
https://github.com/beorn7/perks
Copyright 2013-2015 Blake Mizerany, Björn Rabenstein
See https://github.com/beorn7/perks/blob/master/README.md for license details.
Go support for Protocol Buffers - Google's data interchange format
http://github.com/golang/protobuf/
Copyright 2010 The Go Authors
See source code for license details.

View File

@ -0,0 +1,27 @@
Copyright (c) 2013 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@ -0,0 +1,145 @@
// Copyright 2013 The Go Authors. All rights reserved.
//
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file or at
// https://developers.google.com/open-source/licenses/bsd.
// Package header provides functions for parsing HTTP headers.
package header
import (
"net/http"
"strings"
)
// Octet types from RFC 2616.
var octetTypes [256]octetType
type octetType byte
const (
isToken octetType = 1 << iota
isSpace
)
func init() {
// OCTET = <any 8-bit sequence of data>
// CHAR = <any US-ASCII character (octets 0 - 127)>
// CTL = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
// CR = <US-ASCII CR, carriage return (13)>
// LF = <US-ASCII LF, linefeed (10)>
// SP = <US-ASCII SP, space (32)>
// HT = <US-ASCII HT, horizontal-tab (9)>
// <"> = <US-ASCII double-quote mark (34)>
// CRLF = CR LF
// LWS = [CRLF] 1*( SP | HT )
// TEXT = <any OCTET except CTLs, but including LWS>
// separators = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <">
// | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT
// token = 1*<any CHAR except CTLs or separators>
// qdtext = <any TEXT except <">>
for c := 0; c < 256; c++ {
var t octetType
isCtl := c <= 31 || c == 127
isChar := 0 <= c && c <= 127
isSeparator := strings.ContainsRune(" \t\"(),/:;<=>?@[]\\{}", rune(c))
if strings.ContainsRune(" \t\r\n", rune(c)) {
t |= isSpace
}
if isChar && !isCtl && !isSeparator {
t |= isToken
}
octetTypes[c] = t
}
}
// AcceptSpec describes an Accept* header.
type AcceptSpec struct {
Value string
Q float64
}
// ParseAccept parses Accept* headers.
func ParseAccept(header http.Header, key string) (specs []AcceptSpec) {
loop:
for _, s := range header[key] {
for {
var spec AcceptSpec
spec.Value, s = expectTokenSlash(s)
if spec.Value == "" {
continue loop
}
spec.Q = 1.0
s = skipSpace(s)
if strings.HasPrefix(s, ";") {
s = skipSpace(s[1:])
if !strings.HasPrefix(s, "q=") {
continue loop
}
spec.Q, s = expectQuality(s[2:])
if spec.Q < 0.0 {
continue loop
}
}
specs = append(specs, spec)
s = skipSpace(s)
if !strings.HasPrefix(s, ",") {
continue loop
}
s = skipSpace(s[1:])
}
}
return specs
}
func skipSpace(s string) (rest string) {
i := 0
for ; i < len(s); i++ {
if octetTypes[s[i]]&isSpace == 0 {
break
}
}
return s[i:]
}
func expectTokenSlash(s string) (token, rest string) {
i := 0
for ; i < len(s); i++ {
b := s[i]
if (octetTypes[b]&isToken == 0) && b != '/' {
break
}
}
return s[:i], s[i:]
}
func expectQuality(s string) (q float64, rest string) {
switch {
case len(s) == 0:
return -1, ""
case s[0] == '0':
q = 0
case s[0] == '1':
q = 1
default:
return -1, ""
}
s = s[1:]
if !strings.HasPrefix(s, ".") {
return q, s
}
s = s[1:]
i := 0
n := 0
d := 1
for ; i < len(s); i++ {
b := s[i]
if b < '0' || b > '9' {
break
}
n = n*10 + int(b) - '0'
d *= 10
}
return q + float64(n)/float64(d), s[i:]
}

View File

@ -0,0 +1,36 @@
// Copyright 2013 The Go Authors. All rights reserved.
//
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file or at
// https://developers.google.com/open-source/licenses/bsd.
package httputil
import (
"net/http"
"github.com/prometheus/client_golang/internal/github.com/golang/gddo/httputil/header"
)
// NegotiateContentEncoding returns the best offered content encoding for the
// request's Accept-Encoding header. If two offers match with equal weight and
// then the offer earlier in the list is preferred. If no offers are
// acceptable, then "" is returned.
func NegotiateContentEncoding(r *http.Request, offers []string) string {
bestOffer := "identity"
bestQ := -1.0
specs := header.ParseAccept(r.Header, "Accept-Encoding")
for _, offer := range offers {
for _, spec := range specs {
if spec.Q > bestQ &&
(spec.Value == "*" || spec.Value == offer) {
bestQ = spec.Q
bestOffer = offer
}
}
}
if bestQ == 0 {
bestOffer = ""
}
return bestOffer
}

View File

@ -0,0 +1 @@
command-line-arguments.test

View File

@ -0,0 +1 @@
See [![Go Reference](https://pkg.go.dev/badge/github.com/prometheus/client_golang/prometheus.svg)](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus).

View File

@ -0,0 +1,38 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import "runtime/debug"
// NewBuildInfoCollector is the obsolete version of collectors.NewBuildInfoCollector.
// See there for documentation.
//
// Deprecated: Use collectors.NewBuildInfoCollector instead.
func NewBuildInfoCollector() Collector {
path, version, sum := "unknown", "unknown", "unknown"
if bi, ok := debug.ReadBuildInfo(); ok {
path = bi.Main.Path
version = bi.Main.Version
sum = bi.Main.Sum
}
c := &selfCollector{MustNewConstMetric(
NewDesc(
"go_build_info",
"Build information about the main Go module.",
nil, Labels{"path": path, "version": version, "checksum": sum},
),
GaugeValue, 1)}
c.init(c.self)
return c
}

View File

@ -0,0 +1,128 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
// Collector is the interface implemented by anything that can be used by
// Prometheus to collect metrics. A Collector has to be registered for
// collection. See Registerer.Register.
//
// The stock metrics provided by this package (Gauge, Counter, Summary,
// Histogram, Untyped) are also Collectors (which only ever collect one metric,
// namely itself). An implementer of Collector may, however, collect multiple
// metrics in a coordinated fashion and/or create metrics on the fly. Examples
// for collectors already implemented in this library are the metric vectors
// (i.e. collection of multiple instances of the same Metric but with different
// label values) like GaugeVec or SummaryVec, and the ExpvarCollector.
type Collector interface {
// Describe sends the super-set of all possible descriptors of metrics
// collected by this Collector to the provided channel and returns once
// the last descriptor has been sent. The sent descriptors fulfill the
// consistency and uniqueness requirements described in the Desc
// documentation.
//
// It is valid if one and the same Collector sends duplicate
// descriptors. Those duplicates are simply ignored. However, two
// different Collectors must not send duplicate descriptors.
//
// Sending no descriptor at all marks the Collector as “unchecked”,
// i.e. no checks will be performed at registration time, and the
// Collector may yield any Metric it sees fit in its Collect method.
//
// This method idempotently sends the same descriptors throughout the
// lifetime of the Collector. It may be called concurrently and
// therefore must be implemented in a concurrency safe way.
//
// If a Collector encounters an error while executing this method, it
// must send an invalid descriptor (created with NewInvalidDesc) to
// signal the error to the registry.
Describe(chan<- *Desc)
// Collect is called by the Prometheus registry when collecting
// metrics. The implementation sends each collected metric via the
// provided channel and returns once the last metric has been sent. The
// descriptor of each sent metric is one of those returned by Describe
// (unless the Collector is unchecked, see above). Returned metrics that
// share the same descriptor must differ in their variable label
// values.
//
// This method may be called concurrently and must therefore be
// implemented in a concurrency safe way. Blocking occurs at the expense
// of total performance of rendering all registered metrics. Ideally,
// Collector implementations support concurrent readers.
Collect(chan<- Metric)
}
// DescribeByCollect is a helper to implement the Describe method of a custom
// Collector. It collects the metrics from the provided Collector and sends
// their descriptors to the provided channel.
//
// If a Collector collects the same metrics throughout its lifetime, its
// Describe method can simply be implemented as:
//
// func (c customCollector) Describe(ch chan<- *Desc) {
// DescribeByCollect(c, ch)
// }
//
// However, this will not work if the metrics collected change dynamically over
// the lifetime of the Collector in a way that their combined set of descriptors
// changes as well. The shortcut implementation will then violate the contract
// of the Describe method. If a Collector sometimes collects no metrics at all
// (for example vectors like CounterVec, GaugeVec, etc., which only collect
// metrics after a metric with a fully specified label set has been accessed),
// it might even get registered as an unchecked Collector (cf. the Register
// method of the Registerer interface). Hence, only use this shortcut
// implementation of Describe if you are certain to fulfill the contract.
//
// The Collector example demonstrates a use of DescribeByCollect.
func DescribeByCollect(c Collector, descs chan<- *Desc) {
metrics := make(chan Metric)
go func() {
c.Collect(metrics)
close(metrics)
}()
for m := range metrics {
descs <- m.Desc()
}
}
// selfCollector implements Collector for a single Metric so that the Metric
// collects itself. Add it as an anonymous field to a struct that implements
// Metric, and call init with the Metric itself as an argument.
type selfCollector struct {
self Metric
}
// init provides the selfCollector with a reference to the metric it is supposed
// to collect. It is usually called within the factory function to create a
// metric. See example.
func (c *selfCollector) init(self Metric) {
c.self = self
}
// Describe implements Collector.
func (c *selfCollector) Describe(ch chan<- *Desc) {
ch <- c.self.Desc()
}
// Collect implements Collector.
func (c *selfCollector) Collect(ch chan<- Metric) {
ch <- c.self
}
// collectorMetric is a metric that is also a collector.
// Because of selfCollector, most (if not all) Metrics in
// this package are also collectors.
type collectorMetric interface {
Metric
Collector
}

View File

@ -0,0 +1,30 @@
// Copyright 2025 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
// CollectorFunc is a convenient way to implement a Prometheus Collector
// without interface boilerplate.
// This implementation is based on DescribeByCollect method.
// familiarize yourself to it before using.
type CollectorFunc func(chan<- Metric)
// Collect calls the defined CollectorFunc function with the provided Metrics channel
func (f CollectorFunc) Collect(ch chan<- Metric) {
f(ch)
}
// Describe sends the descriptor information using DescribeByCollect
func (f CollectorFunc) Describe(ch chan<- *Desc) {
DescribeByCollect(f, ch)
}

View File

@ -0,0 +1,40 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package collectors provides implementations of prometheus.Collector to
// conveniently collect process and Go-related metrics.
package collectors
import "github.com/prometheus/client_golang/prometheus"
// NewBuildInfoCollector returns a collector collecting a single metric
// "go_build_info" with the constant value 1 and three labels "path", "version",
// and "checksum". Their label values contain the main module path, version, and
// checksum, respectively. The labels will only have meaningful values if the
// binary is built with Go module support and from source code retrieved from
// the source repository (rather than the local file system). This is usually
// accomplished by building from outside of GOPATH, specifying the full address
// of the main package, e.g. "GO111MODULE=on go run
// github.com/prometheus/client_golang/examples/random". If built without Go
// module support, all label values will be "unknown". If built with Go module
// support but using the source code from the local file system, the "path" will
// be set appropriately, but "checksum" will be empty and "version" will be
// "(devel)".
//
// This collector uses only the build information for the main module. See
// https://github.com/povilasv/prommod for an example of a collector for the
// module dependencies.
func NewBuildInfoCollector() prometheus.Collector {
//nolint:staticcheck // Ignore SA1019 until v2.
return prometheus.NewBuildInfoCollector()
}

View File

@ -0,0 +1,119 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package collectors
import (
"database/sql"
"github.com/prometheus/client_golang/prometheus"
)
type dbStatsCollector struct {
db *sql.DB
maxOpenConnections *prometheus.Desc
openConnections *prometheus.Desc
inUseConnections *prometheus.Desc
idleConnections *prometheus.Desc
waitCount *prometheus.Desc
waitDuration *prometheus.Desc
maxIdleClosed *prometheus.Desc
maxIdleTimeClosed *prometheus.Desc
maxLifetimeClosed *prometheus.Desc
}
// NewDBStatsCollector returns a collector that exports metrics about the given *sql.DB.
// See https://golang.org/pkg/database/sql/#DBStats for more information on stats.
func NewDBStatsCollector(db *sql.DB, dbName string) prometheus.Collector {
fqName := func(name string) string {
return "go_sql_" + name
}
return &dbStatsCollector{
db: db,
maxOpenConnections: prometheus.NewDesc(
fqName("max_open_connections"),
"Maximum number of open connections to the database.",
nil, prometheus.Labels{"db_name": dbName},
),
openConnections: prometheus.NewDesc(
fqName("open_connections"),
"The number of established connections both in use and idle.",
nil, prometheus.Labels{"db_name": dbName},
),
inUseConnections: prometheus.NewDesc(
fqName("in_use_connections"),
"The number of connections currently in use.",
nil, prometheus.Labels{"db_name": dbName},
),
idleConnections: prometheus.NewDesc(
fqName("idle_connections"),
"The number of idle connections.",
nil, prometheus.Labels{"db_name": dbName},
),
waitCount: prometheus.NewDesc(
fqName("wait_count_total"),
"The total number of connections waited for.",
nil, prometheus.Labels{"db_name": dbName},
),
waitDuration: prometheus.NewDesc(
fqName("wait_duration_seconds_total"),
"The total time blocked waiting for a new connection.",
nil, prometheus.Labels{"db_name": dbName},
),
maxIdleClosed: prometheus.NewDesc(
fqName("max_idle_closed_total"),
"The total number of connections closed due to SetMaxIdleConns.",
nil, prometheus.Labels{"db_name": dbName},
),
maxIdleTimeClosed: prometheus.NewDesc(
fqName("max_idle_time_closed_total"),
"The total number of connections closed due to SetConnMaxIdleTime.",
nil, prometheus.Labels{"db_name": dbName},
),
maxLifetimeClosed: prometheus.NewDesc(
fqName("max_lifetime_closed_total"),
"The total number of connections closed due to SetConnMaxLifetime.",
nil, prometheus.Labels{"db_name": dbName},
),
}
}
// Describe implements Collector.
func (c *dbStatsCollector) Describe(ch chan<- *prometheus.Desc) {
ch <- c.maxOpenConnections
ch <- c.openConnections
ch <- c.inUseConnections
ch <- c.idleConnections
ch <- c.waitCount
ch <- c.waitDuration
ch <- c.maxIdleClosed
ch <- c.maxLifetimeClosed
ch <- c.maxIdleTimeClosed
}
// Collect implements Collector.
func (c *dbStatsCollector) Collect(ch chan<- prometheus.Metric) {
stats := c.db.Stats()
ch <- prometheus.MustNewConstMetric(c.maxOpenConnections, prometheus.GaugeValue, float64(stats.MaxOpenConnections))
ch <- prometheus.MustNewConstMetric(c.openConnections, prometheus.GaugeValue, float64(stats.OpenConnections))
ch <- prometheus.MustNewConstMetric(c.inUseConnections, prometheus.GaugeValue, float64(stats.InUse))
ch <- prometheus.MustNewConstMetric(c.idleConnections, prometheus.GaugeValue, float64(stats.Idle))
ch <- prometheus.MustNewConstMetric(c.waitCount, prometheus.CounterValue, float64(stats.WaitCount))
ch <- prometheus.MustNewConstMetric(c.waitDuration, prometheus.CounterValue, stats.WaitDuration.Seconds())
ch <- prometheus.MustNewConstMetric(c.maxIdleClosed, prometheus.CounterValue, float64(stats.MaxIdleClosed))
ch <- prometheus.MustNewConstMetric(c.maxLifetimeClosed, prometheus.CounterValue, float64(stats.MaxLifetimeClosed))
ch <- prometheus.MustNewConstMetric(c.maxIdleTimeClosed, prometheus.CounterValue, float64(stats.MaxIdleTimeClosed))
}

View File

@ -0,0 +1,57 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package collectors
import "github.com/prometheus/client_golang/prometheus"
// NewExpvarCollector returns a newly allocated expvar Collector.
//
// An expvar Collector collects metrics from the expvar interface. It provides a
// quick way to expose numeric values that are already exported via expvar as
// Prometheus metrics. Note that the data models of expvar and Prometheus are
// fundamentally different, and that the expvar Collector is inherently slower
// than native Prometheus metrics. Thus, the expvar Collector is probably great
// for experiments and prototyping, but you should seriously consider a more
// direct implementation of Prometheus metrics for monitoring production
// systems.
//
// The exports map has the following meaning:
//
// The keys in the map correspond to expvar keys, i.e. for every expvar key you
// want to export as Prometheus metric, you need an entry in the exports
// map. The descriptor mapped to each key describes how to export the expvar
// value. It defines the name and the help string of the Prometheus metric
// proxying the expvar value. The type will always be Untyped.
//
// For descriptors without variable labels, the expvar value must be a number or
// a bool. The number is then directly exported as the Prometheus sample
// value. (For a bool, 'false' translates to 0 and 'true' to 1). Expvar values
// that are not numbers or bools are silently ignored.
//
// If the descriptor has one variable label, the expvar value must be an expvar
// map. The keys in the expvar map become the various values of the one
// Prometheus label. The values in the expvar map must be numbers or bools again
// as above.
//
// For descriptors with more than one variable label, the expvar must be a
// nested expvar map, i.e. where the values of the topmost map are maps again
// etc. until a depth is reached that corresponds to the number of labels. The
// leaves of that structure must be numbers or bools as above to serve as the
// sample values.
//
// Anything that does not fit into the scheme above is silently ignored.
func NewExpvarCollector(exports map[string]*prometheus.Desc) prometheus.Collector {
//nolint:staticcheck // Ignore SA1019 until v2.
return prometheus.NewExpvarCollector(exports)
}

View File

@ -0,0 +1,167 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build go1.17
// +build go1.17
package collectors
import (
"regexp"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/internal"
)
var (
// MetricsAll allows all the metrics to be collected from Go runtime.
MetricsAll = GoRuntimeMetricsRule{regexp.MustCompile("/.*")}
// MetricsGC allows only GC metrics to be collected from Go runtime.
// e.g. go_gc_cycles_automatic_gc_cycles_total
// NOTE: This does not include new class of "/cpu/classes/gc/..." metrics.
// Use custom metric rule to access those.
MetricsGC = GoRuntimeMetricsRule{regexp.MustCompile(`^/gc/.*`)}
// MetricsMemory allows only memory metrics to be collected from Go runtime.
// e.g. go_memory_classes_heap_free_bytes
MetricsMemory = GoRuntimeMetricsRule{regexp.MustCompile(`^/memory/.*`)}
// MetricsScheduler allows only scheduler metrics to be collected from Go runtime.
// e.g. go_sched_goroutines_goroutines
MetricsScheduler = GoRuntimeMetricsRule{regexp.MustCompile(`^/sched/.*`)}
// MetricsDebug allows only debug metrics to be collected from Go runtime.
// e.g. go_godebug_non_default_behavior_gocachetest_events_total
MetricsDebug = GoRuntimeMetricsRule{regexp.MustCompile(`^/godebug/.*`)}
)
// WithGoCollectorMemStatsMetricsDisabled disables metrics that is gathered in runtime.MemStats structure such as:
//
// go_memstats_alloc_bytes
// go_memstats_alloc_bytes_total
// go_memstats_sys_bytes
// go_memstats_mallocs_total
// go_memstats_frees_total
// go_memstats_heap_alloc_bytes
// go_memstats_heap_sys_bytes
// go_memstats_heap_idle_bytes
// go_memstats_heap_inuse_bytes
// go_memstats_heap_released_bytes
// go_memstats_heap_objects
// go_memstats_stack_inuse_bytes
// go_memstats_stack_sys_bytes
// go_memstats_mspan_inuse_bytes
// go_memstats_mspan_sys_bytes
// go_memstats_mcache_inuse_bytes
// go_memstats_mcache_sys_bytes
// go_memstats_buck_hash_sys_bytes
// go_memstats_gc_sys_bytes
// go_memstats_other_sys_bytes
// go_memstats_next_gc_bytes
//
// so the metrics known from pre client_golang v1.12.0,
//
// NOTE(bwplotka): The above represents runtime.MemStats statistics, but they are
// actually implemented using new runtime/metrics package. (except skipped go_memstats_gc_cpu_fraction
// -- see https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034 for explanation).
//
// Some users might want to disable this on collector level (although you can use scrape relabelling on Prometheus),
// because similar metrics can be now obtained using WithGoCollectorRuntimeMetrics. Note that the semantics of new
// metrics might be different, plus the names can be change over time with different Go version.
//
// NOTE(bwplotka): Changing metric names can be tedious at times as the alerts, recording rules and dashboards have to be adjusted.
// The old metrics are also very useful, with many guides and books written about how to interpret them.
//
// As a result our recommendation would be to stick with MemStats like metrics and enable other runtime/metrics if you are interested
// in advanced insights Go provides. See ExampleGoCollector_WithAdvancedGoMetrics.
func WithGoCollectorMemStatsMetricsDisabled() func(options *internal.GoCollectorOptions) {
return func(o *internal.GoCollectorOptions) {
o.DisableMemStatsLikeMetrics = true
}
}
// GoRuntimeMetricsRule allow enabling and configuring particular group of runtime/metrics.
// TODO(bwplotka): Consider adding ability to adjust buckets.
type GoRuntimeMetricsRule struct {
// Matcher represents RE2 expression will match the runtime/metrics from https://pkg.go.dev/runtime/metrics
// Use `regexp.MustCompile` or `regexp.Compile` to create this field.
Matcher *regexp.Regexp
}
// WithGoCollectorRuntimeMetrics allows enabling and configuring particular group of runtime/metrics.
// See the list of metrics https://pkg.go.dev/runtime/metrics (pick the Go version you use there!).
// You can use this option in repeated manner, which will add new rules. The order of rules is important, the last rule
// that matches particular metrics is applied.
func WithGoCollectorRuntimeMetrics(rules ...GoRuntimeMetricsRule) func(options *internal.GoCollectorOptions) {
rs := make([]internal.GoCollectorRule, len(rules))
for i, r := range rules {
rs[i] = internal.GoCollectorRule{
Matcher: r.Matcher,
}
}
return func(o *internal.GoCollectorOptions) {
o.RuntimeMetricRules = append(o.RuntimeMetricRules, rs...)
}
}
// WithoutGoCollectorRuntimeMetrics allows disabling group of runtime/metrics that you might have added in WithGoCollectorRuntimeMetrics.
// It behaves similarly to WithGoCollectorRuntimeMetrics just with deny-list semantics.
func WithoutGoCollectorRuntimeMetrics(matchers ...*regexp.Regexp) func(options *internal.GoCollectorOptions) {
rs := make([]internal.GoCollectorRule, len(matchers))
for i, m := range matchers {
rs[i] = internal.GoCollectorRule{
Matcher: m,
Deny: true,
}
}
return func(o *internal.GoCollectorOptions) {
o.RuntimeMetricRules = append(o.RuntimeMetricRules, rs...)
}
}
// GoCollectionOption represents Go collection option flag.
// Deprecated.
type GoCollectionOption uint32
const (
// GoRuntimeMemStatsCollection represents the metrics represented by runtime.MemStats structure.
//
// Deprecated: Use WithGoCollectorMemStatsMetricsDisabled() function to disable those metrics in the collector.
GoRuntimeMemStatsCollection GoCollectionOption = 1 << iota
// GoRuntimeMetricsCollection is the new set of metrics represented by runtime/metrics package.
//
// Deprecated: Use WithGoCollectorRuntimeMetrics(GoRuntimeMetricsRule{Matcher: regexp.MustCompile("/.*")})
// function to enable those metrics in the collector.
GoRuntimeMetricsCollection
)
// WithGoCollections allows enabling different collections for Go collector on top of base metrics.
//
// Deprecated: Use WithGoCollectorRuntimeMetrics() and WithGoCollectorMemStatsMetricsDisabled() instead to control metrics.
func WithGoCollections(flags GoCollectionOption) func(options *internal.GoCollectorOptions) {
return func(options *internal.GoCollectorOptions) {
if flags&GoRuntimeMemStatsCollection == 0 {
WithGoCollectorMemStatsMetricsDisabled()(options)
}
if flags&GoRuntimeMetricsCollection != 0 {
WithGoCollectorRuntimeMetrics(GoRuntimeMetricsRule{Matcher: regexp.MustCompile("/.*")})(options)
}
}
}
// NewGoCollector returns a collector that exports metrics about the current Go
// process using debug.GCStats (base metrics) and runtime/metrics (both in MemStats style and new ones).
func NewGoCollector(opts ...func(o *internal.GoCollectorOptions)) prometheus.Collector {
//nolint:staticcheck // Ignore SA1019 until v2.
return prometheus.NewGoCollector(opts...)
}

View File

@ -0,0 +1,56 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package collectors
import "github.com/prometheus/client_golang/prometheus"
// ProcessCollectorOpts defines the behavior of a process metrics collector
// created with NewProcessCollector.
type ProcessCollectorOpts struct {
// PidFn returns the PID of the process the collector collects metrics
// for. It is called upon each collection. By default, the PID of the
// current process is used, as determined on construction time by
// calling os.Getpid().
PidFn func() (int, error)
// If non-empty, each of the collected metrics is prefixed by the
// provided string and an underscore ("_").
Namespace string
// If true, any error encountered during collection is reported as an
// invalid metric (see NewInvalidMetric). Otherwise, errors are ignored
// and the collected metrics will be incomplete. (Possibly, no metrics
// will be collected at all.) While that's usually not desired, it is
// appropriate for the common "mix-in" of process metrics, where process
// metrics are nice to have, but failing to collect them should not
// disrupt the collection of the remaining metrics.
ReportErrors bool
}
// NewProcessCollector returns a collector which exports the current state of
// process metrics including CPU, memory and file descriptor usage as well as
// the process start time. The detailed behavior is defined by the provided
// ProcessCollectorOpts. The zero value of ProcessCollectorOpts creates a
// collector for the current process with an empty namespace string and no error
// reporting.
//
// The collector only works on operating systems with a Linux-style proc
// filesystem and on Microsoft Windows. On other operating systems, it will not
// collect any metrics.
func NewProcessCollector(opts ProcessCollectorOpts) prometheus.Collector {
//nolint:staticcheck // Ignore SA1019 until v2.
return prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{
PidFn: opts.PidFn,
Namespace: opts.Namespace,
ReportErrors: opts.ReportErrors,
})
}

View File

@ -0,0 +1,361 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"errors"
"math"
"sync/atomic"
"time"
dto "github.com/prometheus/client_model/go"
"google.golang.org/protobuf/types/known/timestamppb"
)
// Counter is a Metric that represents a single numerical value that only ever
// goes up. That implies that it cannot be used to count items whose number can
// also go down, e.g. the number of currently running goroutines. Those
// "counters" are represented by Gauges.
//
// A Counter is typically used to count requests served, tasks completed, errors
// occurred, etc.
//
// To create Counter instances, use NewCounter.
type Counter interface {
Metric
Collector
// Inc increments the counter by 1. Use Add to increment it by arbitrary
// non-negative values.
Inc()
// Add adds the given value to the counter. It panics if the value is <
// 0.
Add(float64)
}
// ExemplarAdder is implemented by Counters that offer the option of adding a
// value to the Counter together with an exemplar. Its AddWithExemplar method
// works like the Add method of the Counter interface but also replaces the
// currently saved exemplar (if any) with a new one, created from the provided
// value, the current time as timestamp, and the provided labels. Empty Labels
// will lead to a valid (label-less) exemplar. But if Labels is nil, the current
// exemplar is left in place. AddWithExemplar panics if the value is < 0, if any
// of the provided labels are invalid, or if the provided labels contain more
// than 128 runes in total.
type ExemplarAdder interface {
AddWithExemplar(value float64, exemplar Labels)
}
// CounterOpts is an alias for Opts. See there for doc comments.
type CounterOpts Opts
// CounterVecOpts bundles the options to create a CounterVec metric.
// It is mandatory to set CounterOpts, see there for mandatory fields. VariableLabels
// is optional and can safely be left to its default value.
type CounterVecOpts struct {
CounterOpts
// VariableLabels are used to partition the metric vector by the given set
// of labels. Each label value will be constrained with the optional Constraint
// function, if provided.
VariableLabels ConstrainableLabels
}
// NewCounter creates a new Counter based on the provided CounterOpts.
//
// The returned implementation also implements ExemplarAdder. It is safe to
// perform the corresponding type assertion.
//
// The returned implementation tracks the counter value in two separate
// variables, a float64 and a uint64. The latter is used to track calls of the
// Inc method and calls of the Add method with a value that can be represented
// as a uint64. This allows atomic increments of the counter with optimal
// performance. (It is common to have an Inc call in very hot execution paths.)
// Both internal tracking values are added up in the Write method. This has to
// be taken into account when it comes to precision and overflow behavior.
func NewCounter(opts CounterOpts) Counter {
desc := V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
UnconstrainedLabels(nil),
opts.ConstLabels,
WithUnit(opts.Unit),
)
if opts.now == nil {
opts.now = time.Now
}
result := &counter{desc: desc, labelPairs: desc.constLabelPairs, now: opts.now}
result.init(result) // Init self-collection.
result.createdTs = timestamppb.New(opts.now())
return result
}
type counter struct {
// valBits contains the bits of the represented float64 value, while
// valInt stores values that are exact integers. Both have to go first
// in the struct to guarantee alignment for atomic operations.
// http://golang.org/pkg/sync/atomic/#pkg-note-BUG
valBits uint64
valInt uint64
selfCollector
desc *Desc
createdTs *timestamppb.Timestamp
labelPairs []*dto.LabelPair
exemplar atomic.Value // Containing nil or a *dto.Exemplar.
// now is for testing purposes, by default it's time.Now.
now func() time.Time
}
func (c *counter) Desc() *Desc {
return c.desc
}
func (c *counter) Add(v float64) {
if v < 0 {
panic(errors.New("counter cannot decrease in value"))
}
ival := uint64(v)
if float64(ival) == v {
atomic.AddUint64(&c.valInt, ival)
return
}
for {
oldBits := atomic.LoadUint64(&c.valBits)
newBits := math.Float64bits(math.Float64frombits(oldBits) + v)
if atomic.CompareAndSwapUint64(&c.valBits, oldBits, newBits) {
return
}
}
}
func (c *counter) AddWithExemplar(v float64, e Labels) {
c.Add(v)
c.updateExemplar(v, e)
}
func (c *counter) Inc() {
atomic.AddUint64(&c.valInt, 1)
}
func (c *counter) get() float64 {
fval := math.Float64frombits(atomic.LoadUint64(&c.valBits))
ival := atomic.LoadUint64(&c.valInt)
return fval + float64(ival)
}
func (c *counter) Write(out *dto.Metric) error {
// Read the Exemplar first and the value second. This is to avoid a race condition
// where users see an exemplar for a not-yet-existing observation.
var exemplar *dto.Exemplar
if e := c.exemplar.Load(); e != nil {
exemplar = e.(*dto.Exemplar)
}
val := c.get()
return populateMetric(CounterValue, val, c.labelPairs, exemplar, out, c.createdTs)
}
func (c *counter) updateExemplar(v float64, l Labels) {
if l == nil {
return
}
e, err := newExemplar(v, c.now(), l)
if err != nil {
panic(err)
}
c.exemplar.Store(e)
}
// CounterVec is a Collector that bundles a set of Counters that all share the
// same Desc, but have different values for their variable labels. This is used
// if you want to count the same thing partitioned by various dimensions
// (e.g. number of HTTP requests, partitioned by response code and
// method). Create instances with NewCounterVec.
type CounterVec struct {
*MetricVec
}
// NewCounterVec creates a new CounterVec based on the provided CounterOpts and
// partitioned by the given label names.
func NewCounterVec(opts CounterOpts, labelNames []string) *CounterVec {
return V2.NewCounterVec(CounterVecOpts{
CounterOpts: opts,
VariableLabels: UnconstrainedLabels(labelNames),
})
}
// NewCounterVec creates a new CounterVec based on the provided CounterVecOpts.
func (v2) NewCounterVec(opts CounterVecOpts) *CounterVec {
desc := V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
opts.VariableLabels,
opts.ConstLabels,
WithUnit(opts.Unit),
)
if opts.now == nil {
opts.now = time.Now
}
return &CounterVec{
MetricVec: NewMetricVec(desc, func(lvs ...string) Metric {
if len(lvs) != len(desc.variableLabels.names) {
panic(makeInconsistentCardinalityError(desc.fqName, desc.variableLabels.names, lvs))
}
result := &counter{desc: desc, labelPairs: MakeLabelPairs(desc, lvs), now: opts.now}
result.init(result) // Init self-collection.
result.createdTs = timestamppb.New(opts.now())
return result
}),
}
}
// GetMetricWithLabelValues returns the Counter for the given slice of label
// values (same order as the variable labels in Desc). If that combination of
// label values is accessed for the first time, a new Counter is created.
//
// It is possible to call this method without using the returned Counter to only
// create the new Counter but leave it at its starting value 0. See also the
// SummaryVec example.
//
// Keeping the Counter for later use is possible (and should be considered if
// performance is critical), but keep in mind that Reset, DeleteLabelValues and
// Delete can be used to delete the Counter from the CounterVec. In that case,
// the Counter will still exist, but it will not be exported anymore, even if a
// Counter with the same label values is created later.
//
// An error is returned if the number of label values is not the same as the
// number of variable labels in Desc (minus any curried labels).
//
// Note that for more than one label value, this method is prone to mistakes
// caused by an incorrect order of arguments. Consider GetMetricWith(Labels) as
// an alternative to avoid that type of mistake. For higher label numbers, the
// latter has a much more readable (albeit more verbose) syntax, but it comes
// with a performance overhead (for creating and processing the Labels map).
// See also the GaugeVec example.
func (v *CounterVec) GetMetricWithLabelValues(lvs ...string) (Counter, error) {
metric, err := v.MetricVec.GetMetricWithLabelValues(lvs...)
if metric != nil {
return metric.(Counter), err
}
return nil, err
}
// GetMetricWith returns the Counter for the given Labels map (the label names
// must match those of the variable labels in Desc). If that label map is
// accessed for the first time, a new Counter is created. Implications of
// creating a Counter without using it and keeping the Counter for later use are
// the same as for GetMetricWithLabelValues.
//
// An error is returned if the number and names of the Labels are inconsistent
// with those of the variable labels in Desc (minus any curried labels).
//
// This method is used for the same purpose as
// GetMetricWithLabelValues(...string). See there for pros and cons of the two
// methods.
func (v *CounterVec) GetMetricWith(labels Labels) (Counter, error) {
metric, err := v.MetricVec.GetMetricWith(labels)
if metric != nil {
return metric.(Counter), err
}
return nil, err
}
// WithLabelValues works as GetMetricWithLabelValues, but panics where
// GetMetricWithLabelValues would have returned an error. Not returning an
// error allows shortcuts like
//
// myVec.WithLabelValues("404", "GET").Add(42)
func (v *CounterVec) WithLabelValues(lvs ...string) Counter {
c, err := v.GetMetricWithLabelValues(lvs...)
if err != nil {
panic(err)
}
return c
}
// With works as GetMetricWith, but panics where GetMetricWithLabels would have
// returned an error. Not returning an error allows shortcuts like
//
// myVec.With(prometheus.Labels{"code": "404", "method": "GET"}).Add(42)
func (v *CounterVec) With(labels Labels) Counter {
c, err := v.GetMetricWith(labels)
if err != nil {
panic(err)
}
return c
}
// CurryWith returns a vector curried with the provided labels, i.e. the
// returned vector has those labels pre-set for all labeled operations performed
// on it. The cardinality of the curried vector is reduced accordingly. The
// order of the remaining labels stays the same (just with the curried labels
// taken out of the sequence which is relevant for the
// (GetMetric)WithLabelValues methods). It is possible to curry a curried
// vector, but only with labels not yet used for currying before.
//
// The metrics contained in the CounterVec are shared between the curried and
// uncurried vectors. They are just accessed differently. Curried and uncurried
// vectors behave identically in terms of collection. Only one must be
// registered with a given registry (usually the uncurried version). The Reset
// method deletes all metrics, even if called on a curried vector.
func (v *CounterVec) CurryWith(labels Labels) (*CounterVec, error) {
vec, err := v.MetricVec.CurryWith(labels)
if vec != nil {
return &CounterVec{vec}, err
}
return nil, err
}
// MustCurryWith works as CurryWith but panics where CurryWith would have
// returned an error.
func (v *CounterVec) MustCurryWith(labels Labels) *CounterVec {
vec, err := v.CurryWith(labels)
if err != nil {
panic(err)
}
return vec
}
// CounterFunc is a Counter whose value is determined at collect time by calling a
// provided function.
//
// To create CounterFunc instances, use NewCounterFunc.
type CounterFunc interface {
Metric
Collector
}
// NewCounterFunc creates a new CounterFunc based on the provided
// CounterOpts. The value reported is determined by calling the given function
// from within the Write method. Take into account that metric collection may
// happen concurrently. If that results in concurrent calls to Write, like in
// the case where a CounterFunc is directly registered with Prometheus, the
// provided function must be concurrency-safe. The function should also honor
// the contract for a Counter (values only go up, not down), but compliance will
// not be checked.
//
// Check out the ExampleGaugeFunc examples for the similar GaugeFunc.
func NewCounterFunc(opts CounterOpts, function func() float64) CounterFunc {
return newValueFunc(V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
UnconstrainedLabels(nil),
opts.ConstLabels,
WithUnit(opts.Unit),
), CounterValue, function)
}

View File

@ -0,0 +1,238 @@
// Copyright 2016 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"fmt"
"sort"
"strings"
"github.com/cespare/xxhash/v2"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/model"
"google.golang.org/protobuf/proto"
"github.com/prometheus/client_golang/prometheus/internal"
)
// Desc is the descriptor used by every Prometheus Metric. It is essentially
// the immutable meta-data of a Metric. The normal Metric implementations
// included in this package manage their Desc under the hood. Users only have to
// deal with Desc if they use advanced features like the ExpvarCollector or
// custom Collectors and Metrics.
//
// Descriptors registered with the same registry have to fulfill certain
// consistency and uniqueness criteria if they share the same fully-qualified
// name: They must have the same help string and the same label names (aka label
// dimensions) in each, constLabels and variableLabels, but they must differ in
// the values of the constLabels.
//
// Descriptors that share the same fully-qualified names and the same label
// values of their constLabels are considered equal.
//
// Use NewDesc to create new Desc instances.
type Desc struct {
// fqName has been built from Namespace, Subsystem, and Name.
fqName string
// help provides some helpful information about this metric.
help string
// unit provides the unit of this metric.
unit string
// constLabelPairs contains precalculated DTO label pairs based on
// the constant labels.
constLabelPairs []*dto.LabelPair
// variableLabels contains names of labels and normalization function for
// which the metric maintains variable values.
variableLabels *compiledLabels
// id is a hash of the values of the ConstLabels and fqName. This
// must be unique among all registered descriptors and can therefore be
// used as an identifier of the descriptor.
id uint64
// dimHash is a hash of the label names (preset and variable) and the
// Help string. Each Desc with the same fqName must have the same
// dimHash.
dimHash uint64
// err is an error that occurred during construction. It is reported on
// registration time.
err error
}
// DescOpt allows setting optional fields for NewDesc.
type DescOpt func(*Desc)
// WithUnit sets the unit for a Desc.
func WithUnit(unit string) DescOpt {
return func(d *Desc) {
d.unit = unit
}
}
// NewDesc allocates and initializes a new Desc. Errors are recorded in the Desc
// and will be reported on registration time. variableLabels and constLabels can
// be nil if no such labels should be set. fqName must not be empty.
//
// variableLabels only contain the label names. Their label values are variable
// and therefore not part of the Desc. (They are managed within the Metric.)
//
// For constLabels, the label values are constant. Therefore, they are fully
// specified in the Desc. See the Collector example for a usage pattern.
func NewDesc(fqName, help string, variableLabels []string, constLabels Labels) *Desc {
return V2.NewDesc(fqName, help, UnconstrainedLabels(variableLabels), constLabels)
}
// NewDesc allocates and initializes a new Desc. Errors are recorded in the Desc
// and will be reported on registration time. variableLabels and constLabels can
// be nil if no such labels should be set. fqName must not be empty.
//
// variableLabels only contain the label names and normalization functions. Their
// label values are variable and therefore not part of the Desc. (They are managed
// within the Metric.)
//
// For constLabels, the label values are constant. Therefore, they are fully
// specified in the Desc. See the Collector example for a usage pattern.
func (v2) NewDesc(fqName, help string, variableLabels ConstrainableLabels, constLabels Labels, opts ...DescOpt) *Desc {
d := &Desc{
fqName: fqName,
help: help,
variableLabels: variableLabels.compile(),
}
for _, opt := range opts {
opt(d)
}
if !model.UTF8Validation.IsValidMetricName(fqName) {
d.err = fmt.Errorf("%q is not a valid metric name", fqName)
return d
}
// labelValues contains the label values of const labels (in order of
// their sorted label names) plus the fqName (at position 0).
labelValues := make([]string, 1, len(constLabels)+1)
labelValues[0] = fqName
labelNames := make([]string, 0, len(constLabels)+len(d.variableLabels.names))
labelNameSet := map[string]struct{}{}
// First add only the const label names and sort them...
for labelName := range constLabels {
if !checkLabelName(labelName) {
d.err = fmt.Errorf("%q is not a valid label name for metric %q", labelName, fqName)
return d
}
labelNames = append(labelNames, labelName)
labelNameSet[labelName] = struct{}{}
}
sort.Strings(labelNames)
// ... so that we can now add const label values in the order of their names.
for _, labelName := range labelNames {
labelValues = append(labelValues, constLabels[labelName])
}
// Validate the const label values. They can't have a wrong cardinality, so
// use in len(labelValues) as expectedNumberOfValues.
if err := validateLabelValues(labelValues, len(labelValues)); err != nil {
d.err = err
return d
}
// Now add the variable label names, but prefix them with something that
// cannot be in a regular label name. That prevents matching the label
// dimension with a different mix between preset and variable labels.
for _, label := range d.variableLabels.names {
if !checkLabelName(label) {
d.err = fmt.Errorf("%q is not a valid label name for metric %q", label, fqName)
return d
}
labelNames = append(labelNames, "$"+label)
labelNameSet[label] = struct{}{}
}
if len(labelNames) != len(labelNameSet) {
d.err = fmt.Errorf("duplicate label names in constant and variable labels for metric %q", fqName)
return d
}
xxh := xxhash.New()
for _, val := range labelValues {
xxh.WriteString(val)
xxh.Write(separatorByteSlice)
}
d.id = xxh.Sum64()
// Sort labelNames so that order doesn't matter for the hash.
sort.Strings(labelNames)
// Now hash together (in this order) the help string, the unit string and the sorted
// label names.
xxh.Reset()
xxh.WriteString(help)
xxh.Write(separatorByteSlice)
xxh.WriteString(d.unit)
xxh.Write(separatorByteSlice)
for _, labelName := range labelNames {
xxh.WriteString(labelName)
xxh.Write(separatorByteSlice)
}
d.dimHash = xxh.Sum64()
d.constLabelPairs = make([]*dto.LabelPair, 0, len(constLabels))
for n, v := range constLabels {
d.constLabelPairs = append(d.constLabelPairs, &dto.LabelPair{
Name: proto.String(n),
Value: proto.String(v),
})
}
sort.Sort(internal.LabelPairSorter(d.constLabelPairs))
return d
}
// NewInvalidDesc returns an invalid descriptor, i.e. a descriptor with the
// provided error set. If a collector returning such a descriptor is registered,
// registration will fail with the provided error. NewInvalidDesc can be used by
// a Collector to signal inability to describe itself.
func NewInvalidDesc(err error) *Desc {
return &Desc{
err: err,
}
}
// Err returns an error that occurred during construction, if any.
//
// Calling this method is optional. It can be used to detect construction
// errors early, before invoking other methods on the Desc. If an error is
// present, later operations may not behave as expected.
func (d *Desc) Err() error {
return d.err
}
func (d *Desc) String() string {
lpStrings := make([]string, 0, len(d.constLabelPairs))
for _, lp := range d.constLabelPairs {
lpStrings = append(
lpStrings,
fmt.Sprintf("%s=%q", lp.GetName(), lp.GetValue()),
)
}
vlStrings := []string{}
if d.variableLabels != nil {
vlStrings = make([]string, 0, len(d.variableLabels.names))
for _, vl := range d.variableLabels.names {
if fn, ok := d.variableLabels.labelConstraints[vl]; ok && fn != nil {
vlStrings = append(vlStrings, fmt.Sprintf("c(%s)", vl))
} else {
vlStrings = append(vlStrings, vl)
}
}
}
return fmt.Sprintf(
"Desc{fqName: %q, help: %q, unit: %q, constLabels: {%s}, variableLabels: {%s}}",
d.fqName,
d.help,
d.unit,
strings.Join(lpStrings, ","),
strings.Join(vlStrings, ","),
)
}

View File

@ -0,0 +1,210 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package prometheus is the core instrumentation package. It provides metrics
// primitives to instrument code for monitoring. It also offers a registry for
// metrics. Sub-packages allow to expose the registered metrics via HTTP
// (package promhttp) or push them to a Pushgateway (package push). There is
// also a sub-package promauto, which provides metrics constructors with
// automatic registration.
//
// All exported functions and methods are safe to be used concurrently unless
// specified otherwise.
//
// # A Basic Example
//
// As a starting point, a very basic usage example:
//
// package main
//
// import (
// "log"
// "net/http"
//
// "github.com/prometheus/client_golang/prometheus"
// "github.com/prometheus/client_golang/prometheus/promhttp"
// )
//
// type metrics struct {
// cpuTemp prometheus.Gauge
// hdFailures *prometheus.CounterVec
// }
//
// func NewMetrics(reg prometheus.Registerer) *metrics {
// m := &metrics{
// cpuTemp: prometheus.NewGauge(prometheus.GaugeOpts{
// Name: "cpu_temperature_celsius",
// Help: "Current temperature of the CPU.",
// }),
// hdFailures: prometheus.NewCounterVec(
// prometheus.CounterOpts{
// Name: "hd_errors_total",
// Help: "Number of hard-disk errors.",
// },
// []string{"device"},
// ),
// }
// reg.MustRegister(m.cpuTemp)
// reg.MustRegister(m.hdFailures)
// return m
// }
//
// func main() {
// // Create a non-global registry.
// reg := prometheus.NewRegistry()
//
// // Create new metrics and register them using the custom registry.
// m := NewMetrics(reg)
// // Set values for the new created metrics.
// m.cpuTemp.Set(65.3)
// m.hdFailures.With(prometheus.Labels{"device":"/dev/sda"}).Inc()
//
// // Expose metrics and custom registry via an HTTP server
// // using the HandleFor function. "/metrics" is the usual endpoint for that.
// http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
// log.Fatal(http.ListenAndServe(":8080", nil))
// }
//
// This is a complete program that exports two metrics, a Gauge and a Counter,
// the latter with a label attached to turn it into a (one-dimensional) vector.
// It register the metrics using a custom registry and exposes them via an HTTP server
// on the /metrics endpoint.
//
// # Metrics
//
// The number of exported identifiers in this package might appear a bit
// overwhelming. However, in addition to the basic plumbing shown in the example
// above, you only need to understand the different metric types and their
// vector versions for basic usage. Furthermore, if you are not concerned with
// fine-grained control of when and how to register metrics with the registry,
// have a look at the promauto package, which will effectively allow you to
// ignore registration altogether in simple cases.
//
// Above, you have already touched the Counter and the Gauge. There are two more
// advanced metric types: the Summary and Histogram. A more thorough description
// of those four metric types can be found in the Prometheus docs:
// https://prometheus.io/docs/concepts/metric_types/
//
// In addition to the fundamental metric types Gauge, Counter, Summary, and
// Histogram, a very important part of the Prometheus data model is the
// partitioning of samples along dimensions called labels, which results in
// metric vectors. The fundamental types are GaugeVec, CounterVec, SummaryVec,
// and HistogramVec.
//
// While only the fundamental metric types implement the Metric interface, both
// the metrics and their vector versions implement the Collector interface. A
// Collector manages the collection of a number of Metrics, but for convenience,
// a Metric can also “collect itself”. Note that Gauge, Counter, Summary, and
// Histogram are interfaces themselves while GaugeVec, CounterVec, SummaryVec,
// and HistogramVec are not.
//
// To create instances of Metrics and their vector versions, you need a suitable
// …Opts struct, i.e. GaugeOpts, CounterOpts, SummaryOpts, or HistogramOpts.
//
// # Custom Collectors and constant Metrics
//
// While you could create your own implementations of Metric, most likely you
// will only ever implement the Collector interface on your own. At a first
// glance, a custom Collector seems handy to bundle Metrics for common
// registration (with the prime example of the different metric vectors above,
// which bundle all the metrics of the same name but with different labels).
//
// There is a more involved use case, too: If you already have metrics
// available, created outside of the Prometheus context, you don't need the
// interface of the various Metric types. You essentially want to mirror the
// existing numbers into Prometheus Metrics during collection. An own
// implementation of the Collector interface is perfect for that. You can create
// Metric instances “on the fly” using NewConstMetric, NewConstHistogram, and
// NewConstSummary (and their respective Must… versions). NewConstMetric is used
// for all metric types with just a float64 as their value: Counter, Gauge, and
// a special “type” called Untyped. Use the latter if you are not sure if the
// mirrored metric is a Counter or a Gauge. Creation of the Metric instance
// happens in the Collect method. The Describe method has to return separate
// Desc instances, representative of the “throw-away” metrics to be created
// later. NewDesc comes in handy to create those Desc instances. Alternatively,
// you could return no Desc at all, which will mark the Collector “unchecked”.
// No checks are performed at registration time, but metric consistency will
// still be ensured at scrape time, i.e. any inconsistencies will lead to scrape
// errors. Thus, with unchecked Collectors, the responsibility to not collect
// metrics that lead to inconsistencies in the total scrape result lies with the
// implementer of the Collector. While this is not a desirable state, it is
// sometimes necessary. The typical use case is a situation where the exact
// metrics to be returned by a Collector cannot be predicted at registration
// time, but the implementer has sufficient knowledge of the whole system to
// guarantee metric consistency.
//
// The Collector example illustrates the use case. You can also look at the
// source code of the processCollector (mirroring process metrics), the
// goCollector (mirroring Go metrics), or the expvarCollector (mirroring expvar
// metrics) as examples that are used in this package itself.
//
// If you just need to call a function to get a single float value to collect as
// a metric, GaugeFunc, CounterFunc, or UntypedFunc might be interesting
// shortcuts.
//
// # Advanced Uses of the Registry
//
// While MustRegister is the by far most common way of registering a Collector,
// sometimes you might want to handle the errors the registration might cause.
// As suggested by the name, MustRegister panics if an error occurs. With the
// Register function, the error is returned and can be handled.
//
// An error is returned if the registered Collector is incompatible or
// inconsistent with already registered metrics. The registry aims for
// consistency of the collected metrics according to the Prometheus data model.
// Inconsistencies are ideally detected at registration time, not at collect
// time. The former will usually be detected at start-up time of a program,
// while the latter will only happen at scrape time, possibly not even on the
// first scrape if the inconsistency only becomes relevant later. That is the
// main reason why a Collector and a Metric have to describe themselves to the
// registry.
//
// So far, everything we did operated on the so-called default registry, as it
// can be found in the global DefaultRegisterer variable. With NewRegistry, you
// can create a custom registry, or you can even implement the Registerer or
// Gatherer interfaces yourself. The methods Register and Unregister work in the
// same way on a custom registry as the global functions Register and Unregister
// on the default registry.
//
// There are a number of uses for custom registries: You can use registries with
// special properties, see NewPedanticRegistry. You can avoid global state, as
// it is imposed by the DefaultRegisterer. You can use multiple registries at
// the same time to expose different metrics in different ways. You can use
// separate registries for testing purposes.
//
// Also note that the DefaultRegisterer comes registered with a Collector for Go
// runtime metrics (via NewGoCollector) and a Collector for process metrics (via
// NewProcessCollector). With a custom registry, you are in control and decide
// yourself about the Collectors to register.
//
// # HTTP Exposition
//
// The Registry implements the Gatherer interface. The caller of the Gather
// method can then expose the gathered metrics in some way. Usually, the metrics
// are served via HTTP on the /metrics endpoint. That's happening in the example
// above. The tools to expose metrics via HTTP are in the promhttp sub-package.
//
// # Pushing to the Pushgateway
//
// Function for pushing to the Pushgateway can be found in the push sub-package.
//
// # Graphite Bridge
//
// Functions and examples to push metrics from a Gatherer to Graphite can be
// found in the graphite sub-package.
//
// # Other Means of Exposition
//
// More ways of exposing metrics can easily be added by following the approaches
// of the existing implementations.
package prometheus

View File

@ -0,0 +1,86 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"encoding/json"
"expvar"
)
type expvarCollector struct {
exports map[string]*Desc
}
// NewExpvarCollector is the obsolete version of collectors.NewExpvarCollector.
// See there for documentation.
//
// Deprecated: Use collectors.NewExpvarCollector instead.
func NewExpvarCollector(exports map[string]*Desc) Collector {
return &expvarCollector{
exports: exports,
}
}
// Describe implements Collector.
func (e *expvarCollector) Describe(ch chan<- *Desc) {
for _, desc := range e.exports {
ch <- desc
}
}
// Collect implements Collector.
func (e *expvarCollector) Collect(ch chan<- Metric) {
for name, desc := range e.exports {
var m Metric
expVar := expvar.Get(name)
if expVar == nil {
continue
}
var v any
labels := make([]string, len(desc.variableLabels.names))
if err := json.Unmarshal([]byte(expVar.String()), &v); err != nil {
ch <- NewInvalidMetric(desc, err)
continue
}
var processValue func(v any, i int)
processValue = func(v any, i int) {
if i >= len(labels) {
copiedLabels := append(make([]string, 0, len(labels)), labels...)
switch v := v.(type) {
case float64:
m = MustNewConstMetric(desc, UntypedValue, v, copiedLabels...)
case bool:
if v {
m = MustNewConstMetric(desc, UntypedValue, 1, copiedLabels...)
} else {
m = MustNewConstMetric(desc, UntypedValue, 0, copiedLabels...)
}
default:
return
}
ch <- m
return
}
vm, ok := v.(map[string]any)
if !ok {
return
}
for lv, val := range vm {
labels[i] = lv
processValue(val, i+1)
}
}
processValue(v, 0)
}
}

View File

@ -0,0 +1,42 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
// Inline and byte-free variant of hash/fnv's fnv64a.
const (
offset64 = 14695981039346656037
prime64 = 1099511628211
)
// hashNew initializies a new fnv64a hash value.
func hashNew() uint64 {
return offset64
}
// hashAdd adds a string to a fnv64a hash value, returning the updated hash.
func hashAdd(h uint64, s string) uint64 {
for i := 0; i < len(s); i++ {
h ^= uint64(s[i])
h *= prime64
}
return h
}
// hashAddByte adds a byte to a fnv64a hash value, returning the updated hash.
func hashAddByte(h uint64, b byte) uint64 {
h ^= uint64(b)
h *= prime64
return h
}

View File

@ -0,0 +1,314 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"math"
"sync/atomic"
"time"
dto "github.com/prometheus/client_model/go"
)
// Gauge is a Metric that represents a single numerical value that can
// arbitrarily go up and down.
//
// A Gauge is typically used for measured values like temperatures or current
// memory usage, but also "counts" that can go up and down, like the number of
// running goroutines.
//
// To create Gauge instances, use NewGauge.
type Gauge interface {
Metric
Collector
// Set sets the Gauge to an arbitrary value.
Set(float64)
// Inc increments the Gauge by 1. Use Add to increment it by arbitrary
// values.
Inc()
// Dec decrements the Gauge by 1. Use Sub to decrement it by arbitrary
// values.
Dec()
// Add adds the given value to the Gauge. (The value can be negative,
// resulting in a decrease of the Gauge.)
Add(float64)
// Sub subtracts the given value from the Gauge. (The value can be
// negative, resulting in an increase of the Gauge.)
Sub(float64)
// SetToCurrentTime sets the Gauge to the current Unix time in seconds.
SetToCurrentTime()
}
// GaugeOpts is an alias for Opts. See there for doc comments.
type GaugeOpts Opts
// GaugeVecOpts bundles the options to create a GaugeVec metric.
// It is mandatory to set GaugeOpts, see there for mandatory fields. VariableLabels
// is optional and can safely be left to its default value.
type GaugeVecOpts struct {
GaugeOpts
// VariableLabels are used to partition the metric vector by the given set
// of labels. Each label value will be constrained with the optional Constraint
// function, if provided.
VariableLabels ConstrainableLabels
}
// NewGauge creates a new Gauge based on the provided GaugeOpts.
//
// The returned implementation is optimized for a fast Set method. If you have a
// choice for managing the value of a Gauge via Set vs. Inc/Dec/Add/Sub, pick
// the former. For example, the Inc method of the returned Gauge is slower than
// the Inc method of a Counter returned by NewCounter. This matches the typical
// scenarios for Gauges and Counters, where the former tends to be Set-heavy and
// the latter Inc-heavy.
func NewGauge(opts GaugeOpts) Gauge {
desc := V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
UnconstrainedLabels(nil),
opts.ConstLabels,
WithUnit(opts.Unit),
)
result := &gauge{desc: desc, labelPairs: desc.constLabelPairs}
result.init(result) // Init self-collection.
return result
}
type gauge struct {
// valBits contains the bits of the represented float64 value. It has
// to go first in the struct to guarantee alignment for atomic
// operations. http://golang.org/pkg/sync/atomic/#pkg-note-BUG
valBits uint64
selfCollector
desc *Desc
labelPairs []*dto.LabelPair
}
func (g *gauge) Desc() *Desc {
return g.desc
}
func (g *gauge) Set(val float64) {
atomic.StoreUint64(&g.valBits, math.Float64bits(val))
}
func (g *gauge) SetToCurrentTime() {
g.Set(float64(time.Now().UnixNano()) / 1e9)
}
func (g *gauge) Inc() {
g.Add(1)
}
func (g *gauge) Dec() {
g.Add(-1)
}
func (g *gauge) Add(val float64) {
for {
oldBits := atomic.LoadUint64(&g.valBits)
newBits := math.Float64bits(math.Float64frombits(oldBits) + val)
if atomic.CompareAndSwapUint64(&g.valBits, oldBits, newBits) {
return
}
}
}
func (g *gauge) Sub(val float64) {
g.Add(val * -1)
}
func (g *gauge) Write(out *dto.Metric) error {
val := math.Float64frombits(atomic.LoadUint64(&g.valBits))
return populateMetric(GaugeValue, val, g.labelPairs, nil, out, nil)
}
// GaugeVec is a Collector that bundles a set of Gauges that all share the same
// Desc, but have different values for their variable labels. This is used if
// you want to count the same thing partitioned by various dimensions
// (e.g. number of operations queued, partitioned by user and operation
// type). Create instances with NewGaugeVec.
type GaugeVec struct {
*MetricVec
}
// NewGaugeVec creates a new GaugeVec based on the provided GaugeOpts and
// partitioned by the given label names.
func NewGaugeVec(opts GaugeOpts, labelNames []string) *GaugeVec {
return V2.NewGaugeVec(GaugeVecOpts{
GaugeOpts: opts,
VariableLabels: UnconstrainedLabels(labelNames),
})
}
// NewGaugeVec creates a new GaugeVec based on the provided GaugeVecOpts.
func (v2) NewGaugeVec(opts GaugeVecOpts) *GaugeVec {
desc := V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
opts.VariableLabels,
opts.ConstLabels,
WithUnit(opts.Unit),
)
return &GaugeVec{
MetricVec: NewMetricVec(desc, func(lvs ...string) Metric {
if len(lvs) != len(desc.variableLabels.names) {
panic(makeInconsistentCardinalityError(desc.fqName, desc.variableLabels.names, lvs))
}
result := &gauge{desc: desc, labelPairs: MakeLabelPairs(desc, lvs)}
result.init(result) // Init self-collection.
return result
}),
}
}
// GetMetricWithLabelValues returns the Gauge for the given slice of label
// values (same order as the variable labels in Desc). If that combination of
// label values is accessed for the first time, a new Gauge is created.
//
// It is possible to call this method without using the returned Gauge to only
// create the new Gauge but leave it at its starting value 0. See also the
// SummaryVec example.
//
// Keeping the Gauge for later use is possible (and should be considered if
// performance is critical), but keep in mind that Reset, DeleteLabelValues and
// Delete can be used to delete the Gauge from the GaugeVec. In that case, the
// Gauge will still exist, but it will not be exported anymore, even if a
// Gauge with the same label values is created later. See also the CounterVec
// example.
//
// An error is returned if the number of label values is not the same as the
// number of variable labels in Desc (minus any curried labels).
//
// Note that for more than one label value, this method is prone to mistakes
// caused by an incorrect order of arguments. Consider GetMetricWith(Labels) as
// an alternative to avoid that type of mistake. For higher label numbers, the
// latter has a much more readable (albeit more verbose) syntax, but it comes
// with a performance overhead (for creating and processing the Labels map).
func (v *GaugeVec) GetMetricWithLabelValues(lvs ...string) (Gauge, error) {
metric, err := v.MetricVec.GetMetricWithLabelValues(lvs...)
if metric != nil {
return metric.(Gauge), err
}
return nil, err
}
// GetMetricWith returns the Gauge for the given Labels map (the label names
// must match those of the variable labels in Desc). If that label map is
// accessed for the first time, a new Gauge is created. Implications of
// creating a Gauge without using it and keeping the Gauge for later use are
// the same as for GetMetricWithLabelValues.
//
// An error is returned if the number and names of the Labels are inconsistent
// with those of the variable labels in Desc (minus any curried labels).
//
// This method is used for the same purpose as
// GetMetricWithLabelValues(...string). See there for pros and cons of the two
// methods.
func (v *GaugeVec) GetMetricWith(labels Labels) (Gauge, error) {
metric, err := v.MetricVec.GetMetricWith(labels)
if metric != nil {
return metric.(Gauge), err
}
return nil, err
}
// WithLabelValues works as GetMetricWithLabelValues, but panics where
// GetMetricWithLabelValues would have returned an error. Not returning an
// error allows shortcuts like
//
// myVec.WithLabelValues("404", "GET").Add(42)
func (v *GaugeVec) WithLabelValues(lvs ...string) Gauge {
g, err := v.GetMetricWithLabelValues(lvs...)
if err != nil {
panic(err)
}
return g
}
// With works as GetMetricWith, but panics where GetMetricWithLabels would have
// returned an error. Not returning an error allows shortcuts like
//
// myVec.With(prometheus.Labels{"code": "404", "method": "GET"}).Add(42)
func (v *GaugeVec) With(labels Labels) Gauge {
g, err := v.GetMetricWith(labels)
if err != nil {
panic(err)
}
return g
}
// CurryWith returns a vector curried with the provided labels, i.e. the
// returned vector has those labels pre-set for all labeled operations performed
// on it. The cardinality of the curried vector is reduced accordingly. The
// order of the remaining labels stays the same (just with the curried labels
// taken out of the sequence which is relevant for the
// (GetMetric)WithLabelValues methods). It is possible to curry a curried
// vector, but only with labels not yet used for currying before.
//
// The metrics contained in the GaugeVec are shared between the curried and
// uncurried vectors. They are just accessed differently. Curried and uncurried
// vectors behave identically in terms of collection. Only one must be
// registered with a given registry (usually the uncurried version). The Reset
// method deletes all metrics, even if called on a curried vector.
func (v *GaugeVec) CurryWith(labels Labels) (*GaugeVec, error) {
vec, err := v.MetricVec.CurryWith(labels)
if vec != nil {
return &GaugeVec{vec}, err
}
return nil, err
}
// MustCurryWith works as CurryWith but panics where CurryWith would have
// returned an error.
func (v *GaugeVec) MustCurryWith(labels Labels) *GaugeVec {
vec, err := v.CurryWith(labels)
if err != nil {
panic(err)
}
return vec
}
// GaugeFunc is a Gauge whose value is determined at collect time by calling a
// provided function.
//
// To create GaugeFunc instances, use NewGaugeFunc.
type GaugeFunc interface {
Metric
Collector
}
// NewGaugeFunc creates a new GaugeFunc based on the provided GaugeOpts. The
// value reported is determined by calling the given function from within the
// Write method. Take into account that metric collection may happen
// concurrently. Therefore, it must be safe to call the provided function
// concurrently.
//
// NewGaugeFunc is a good way to create an “info” style metric with a constant
// value of 1. Example:
// https://github.com/prometheus/common/blob/8558a5b7db3c84fa38b4766966059a7bd5bfa2ee/version/info.go#L36-L56
func NewGaugeFunc(opts GaugeOpts, function func() float64) GaugeFunc {
return newValueFunc(V2.NewDesc(
BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
UnconstrainedLabels(nil),
opts.ConstLabels,
WithUnit(opts.Unit),
), GaugeValue, function)
}

View File

@ -0,0 +1,26 @@
// Copyright 2015 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build !js || wasm
// +build !js wasm
package prometheus
import "os"
func getPIDFn() func() (int, error) {
pid := os.Getpid()
return func() (int, error) {
return pid, nil
}
}

View File

@ -0,0 +1,23 @@
// Copyright 2015 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build js && !wasm
// +build js,!wasm
package prometheus
func getPIDFn() func() (int, error) {
return func() (int, error) {
return 1, nil
}
}

View File

@ -0,0 +1,274 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"runtime"
"runtime/debug"
"time"
)
// goRuntimeMemStats provides the metrics initially provided by runtime.ReadMemStats.
// From Go 1.17 those similar (and better) statistics are provided by runtime/metrics, so
// while eval closure works on runtime.MemStats, the struct from Go 1.17+ is
// populated using runtime/metrics. Those are the defaults we can't alter.
func goRuntimeMemStats() memStatsMetrics {
return memStatsMetrics{
{
desc: NewDesc(
memstatNamespace("alloc_bytes"),
"Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Alloc) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("alloc_bytes_total"),
"Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.TotalAlloc) },
valType: CounterValue,
}, {
desc: NewDesc(
memstatNamespace("sys_bytes"),
"Number of bytes obtained from system. Equals to /memory/classes/total:byte.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Sys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("mallocs_total"),
// TODO(bwplotka): We could add go_memstats_heap_objects, probably useful for discovery. Let's gather more feedback, kind of a waste of bytes for everybody for compatibility reasons to keep both, and we can't really rename/remove useful metric.
"Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Mallocs) },
valType: CounterValue,
}, {
desc: NewDesc(
memstatNamespace("frees_total"),
"Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Frees) },
valType: CounterValue,
}, {
desc: NewDesc(
memstatNamespace("heap_alloc_bytes"),
"Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapAlloc) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("heap_sys_bytes"),
"Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("heap_idle_bytes"),
"Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapIdle) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("heap_inuse_bytes"),
"Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapInuse) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("heap_released_bytes"),
"Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapReleased) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("heap_objects"),
"Number of currently allocated objects. Equals to /gc/heap/objects:objects.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapObjects) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("stack_inuse_bytes"),
"Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackInuse) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("stack_sys_bytes"),
"Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("mspan_inuse_bytes"),
"Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanInuse) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("mspan_sys_bytes"),
"Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("mcache_inuse_bytes"),
"Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheInuse) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("mcache_sys_bytes"),
"Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("buck_hash_sys_bytes"),
"Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.BuckHashSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("gc_sys_bytes"),
"Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.GCSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("other_sys_bytes"),
"Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.OtherSys) },
valType: GaugeValue,
}, {
desc: NewDesc(
memstatNamespace("next_gc_bytes"),
"Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.",
nil, nil,
),
eval: func(ms *runtime.MemStats) float64 { return float64(ms.NextGC) },
valType: GaugeValue,
},
}
}
type baseGoCollector struct {
goroutinesDesc *Desc
threadsDesc *Desc
gcDesc *Desc
gcLastTimeDesc *Desc
goInfoDesc *Desc
}
func newBaseGoCollector() baseGoCollector {
return baseGoCollector{
goroutinesDesc: NewDesc(
"go_goroutines",
"Number of goroutines that currently exist.",
nil, nil),
threadsDesc: NewDesc(
"go_threads",
"Number of OS threads created.",
nil, nil),
gcDesc: NewDesc(
"go_gc_duration_seconds",
"A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.",
nil, nil),
gcLastTimeDesc: NewDesc(
"go_memstats_last_gc_time_seconds",
"Number of seconds since 1970 of last garbage collection.",
nil, nil),
goInfoDesc: NewDesc(
"go_info",
"Information about the Go environment.",
nil, Labels{"version": runtime.Version()}),
}
}
// Describe returns all descriptions of the collector.
func (c *baseGoCollector) Describe(ch chan<- *Desc) {
ch <- c.goroutinesDesc
ch <- c.threadsDesc
ch <- c.gcDesc
ch <- c.gcLastTimeDesc
ch <- c.goInfoDesc
}
// Collect returns the current state of all metrics of the collector.
func (c *baseGoCollector) Collect(ch chan<- Metric) {
ch <- MustNewConstMetric(c.goroutinesDesc, GaugeValue, float64(runtime.NumGoroutine()))
n := getRuntimeNumThreads()
ch <- MustNewConstMetric(c.threadsDesc, GaugeValue, n)
var stats debug.GCStats
stats.PauseQuantiles = make([]time.Duration, 5)
debug.ReadGCStats(&stats)
quantiles := make(map[float64]float64)
for idx, pq := range stats.PauseQuantiles[1:] {
quantiles[float64(idx+1)/float64(len(stats.PauseQuantiles)-1)] = pq.Seconds()
}
quantiles[0.0] = stats.PauseQuantiles[0].Seconds()
ch <- MustNewConstSummary(c.gcDesc, uint64(stats.NumGC), stats.PauseTotal.Seconds(), quantiles)
ch <- MustNewConstMetric(c.gcLastTimeDesc, GaugeValue, float64(stats.LastGC.UnixNano())/1e9)
ch <- MustNewConstMetric(c.goInfoDesc, GaugeValue, 1)
}
func memstatNamespace(s string) string {
return "go_memstats_" + s
}
// memStatsMetrics provide description, evaluator, runtime/metrics name, and
// value type for memstat metrics.
type memStatsMetrics []struct {
desc *Desc
eval func(*runtime.MemStats) float64
valType ValueType
}

View File

@ -0,0 +1,586 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build go1.17
// +build go1.17
package prometheus
import (
"fmt"
"math"
"runtime"
"runtime/metrics"
"strings"
"sync"
"github.com/prometheus/client_golang/prometheus/internal"
dto "github.com/prometheus/client_model/go"
"google.golang.org/protobuf/proto"
)
const (
// constants for strings referenced more than once.
goGCHeapTinyAllocsObjects = "/gc/heap/tiny/allocs:objects"
goGCHeapAllocsObjects = "/gc/heap/allocs:objects"
goGCHeapFreesObjects = "/gc/heap/frees:objects"
goGCHeapFreesBytes = "/gc/heap/frees:bytes"
goGCHeapAllocsBytes = "/gc/heap/allocs:bytes"
goGCHeapObjects = "/gc/heap/objects:objects"
goGCHeapGoalBytes = "/gc/heap/goal:bytes"
goMemoryClassesTotalBytes = "/memory/classes/total:bytes"
goMemoryClassesHeapObjectsBytes = "/memory/classes/heap/objects:bytes"
goMemoryClassesHeapUnusedBytes = "/memory/classes/heap/unused:bytes"
goMemoryClassesHeapReleasedBytes = "/memory/classes/heap/released:bytes"
goMemoryClassesHeapFreeBytes = "/memory/classes/heap/free:bytes"
goMemoryClassesHeapStacksBytes = "/memory/classes/heap/stacks:bytes"
goMemoryClassesOSStacksBytes = "/memory/classes/os-stacks:bytes"
goMemoryClassesMetadataMSpanInuseBytes = "/memory/classes/metadata/mspan/inuse:bytes"
goMemoryClassesMetadataMSPanFreeBytes = "/memory/classes/metadata/mspan/free:bytes"
goMemoryClassesMetadataMCacheInuseBytes = "/memory/classes/metadata/mcache/inuse:bytes"
goMemoryClassesMetadataMCacheFreeBytes = "/memory/classes/metadata/mcache/free:bytes"
goMemoryClassesProfilingBucketsBytes = "/memory/classes/profiling/buckets:bytes"
goMemoryClassesMetadataOtherBytes = "/memory/classes/metadata/other:bytes"
goMemoryClassesOtherBytes = "/memory/classes/other:bytes"
)
// rmNamesForMemStatsMetrics represents runtime/metrics names required to populate goRuntimeMemStats from like logic.
var rmNamesForMemStatsMetrics = []string{
goGCHeapTinyAllocsObjects,
goGCHeapAllocsObjects,
goGCHeapFreesObjects,
goGCHeapAllocsBytes,
goGCHeapObjects,
goGCHeapGoalBytes,
goMemoryClassesTotalBytes,
goMemoryClassesHeapObjectsBytes,
goMemoryClassesHeapUnusedBytes,
goMemoryClassesHeapReleasedBytes,
goMemoryClassesHeapFreeBytes,
goMemoryClassesHeapStacksBytes,
goMemoryClassesOSStacksBytes,
goMemoryClassesMetadataMSpanInuseBytes,
goMemoryClassesMetadataMSPanFreeBytes,
goMemoryClassesMetadataMCacheInuseBytes,
goMemoryClassesMetadataMCacheFreeBytes,
goMemoryClassesProfilingBucketsBytes,
goMemoryClassesMetadataOtherBytes,
goMemoryClassesOtherBytes,
}
func bestEffortLookupRM(lookup []string) []metrics.Description {
ret := make([]metrics.Description, 0, len(lookup))
for _, rm := range metrics.All() {
for _, m := range lookup {
if m == rm.Name {
ret = append(ret, rm)
}
}
}
return ret
}
type goCollector struct {
base baseGoCollector
// mu protects updates to all fields ensuring a consistent
// snapshot is always produced by Collect.
mu sync.Mutex
// Contains all samples that have to be retrieved from runtime/metrics (not all of them will be exposed).
sampleBuf []metrics.Sample
// sampleMap allows lookup for MemStats metrics and runtime/metrics histograms for exact sums.
sampleMap map[string]*metrics.Sample
// rmExposedMetrics represents all runtime/metrics package metrics
// that were configured to be exposed.
rmExposedMetrics []collectorMetric
rmExactSumMapForHist map[string]string
// With Go 1.17, the runtime/metrics package was introduced.
// From that point on, metric names produced by the runtime/metrics
// package could be generated from runtime/metrics names. However,
// these differ from the old names for the same values.
//
// This field exists to export the same values under the old names
// as well.
msMetrics memStatsMetrics
msMetricsEnabled bool
}
type rmMetricDesc struct {
metrics.Description
}
func matchRuntimeMetricsRules(rules []internal.GoCollectorRule) []rmMetricDesc {
var descs []rmMetricDesc
for _, d := range metrics.All() {
var (
deny = true
desc rmMetricDesc
)
for _, r := range rules {
if !r.Matcher.MatchString(d.Name) {
continue
}
deny = r.Deny
}
if deny {
continue
}
desc.Description = d
descs = append(descs, desc)
}
return descs
}
func defaultGoCollectorOptions() internal.GoCollectorOptions {
return internal.GoCollectorOptions{
RuntimeMetricSumForHist: map[string]string{
"/gc/heap/allocs-by-size:bytes": goGCHeapAllocsBytes,
"/gc/heap/frees-by-size:bytes": goGCHeapFreesBytes,
},
RuntimeMetricRules: []internal.GoCollectorRule{
// Recommended metrics we want by default from runtime/metrics.
{Matcher: internal.GoCollectorDefaultRuntimeMetrics},
},
}
}
// NewGoCollector is the obsolete version of collectors.NewGoCollector.
// See there for documentation.
//
// Deprecated: Use collectors.NewGoCollector instead.
func NewGoCollector(opts ...func(o *internal.GoCollectorOptions)) Collector {
opt := defaultGoCollectorOptions()
for _, o := range opts {
o(&opt)
}
exposedDescriptions := matchRuntimeMetricsRules(opt.RuntimeMetricRules)
// Collect all histogram samples so that we can get their buckets.
// The API guarantees that the buckets are always fixed for the lifetime
// of the process.
var histograms []metrics.Sample
for _, d := range exposedDescriptions {
if d.Kind == metrics.KindFloat64Histogram {
histograms = append(histograms, metrics.Sample{Name: d.Name})
}
}
if len(histograms) > 0 {
metrics.Read(histograms)
}
bucketsMap := make(map[string][]float64)
for i := range histograms {
bucketsMap[histograms[i].Name] = histograms[i].Value.Float64Histogram().Buckets
}
// Generate a collector for each exposed runtime/metrics metric.
metricSet := make([]collectorMetric, 0, len(exposedDescriptions))
// SampleBuf is used for reading from runtime/metrics.
// We are assuming the largest case to have stable pointers for sampleMap purposes.
sampleBuf := make([]metrics.Sample, 0, len(exposedDescriptions)+len(opt.RuntimeMetricSumForHist)+len(rmNamesForMemStatsMetrics))
sampleMap := make(map[string]*metrics.Sample, len(exposedDescriptions))
for _, d := range exposedDescriptions {
namespace, subsystem, name, ok := internal.RuntimeMetricsToProm(&d.Description)
if !ok {
// Just ignore this metric; we can't do anything with it here.
// If a user decides to use the latest version of Go, we don't want
// to fail here. This condition is tested in TestExpectedRuntimeMetrics.
continue
}
help := attachOriginalName(d.Description.Description, d.Name)
sampleBuf = append(sampleBuf, metrics.Sample{Name: d.Name})
sampleMap[d.Name] = &sampleBuf[len(sampleBuf)-1]
// Extract unit from the runtime/metrics name (e.g., "/gc/heap/allocs:bytes" -> "bytes")
// and sanitize to match Prometheus naming conventions (e.g., "cpu-seconds" -> "cpu_seconds")
var unit string
if idx := strings.IndexRune(d.Name, ':'); idx >= 0 {
unit = d.Name[idx+1:]
unit = strings.ReplaceAll(unit, "-", "_")
unit = strings.ReplaceAll(unit, "*", "_")
unit = strings.ReplaceAll(unit, "/", "_per_")
}
var m collectorMetric
if d.Kind == metrics.KindFloat64Histogram {
_, hasSum := opt.RuntimeMetricSumForHist[d.Name]
m = newBatchHistogram(
V2.NewDesc(
BuildFQName(namespace, subsystem, name),
help,
UnconstrainedLabels(nil),
nil,
WithUnit(unit),
),
internal.RuntimeMetricsBucketsForUnit(bucketsMap[d.Name], unit),
hasSum,
)
} else if d.Cumulative {
m = NewCounter(CounterOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: name,
Help: help,
Unit: unit,
},
)
} else {
m = NewGauge(GaugeOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: name,
Help: help,
Unit: unit,
})
}
metricSet = append(metricSet, m)
}
// Add exact sum metrics to sampleBuf if not added before.
for _, h := range histograms {
sumMetric, ok := opt.RuntimeMetricSumForHist[h.Name]
if !ok {
continue
}
if _, ok := sampleMap[sumMetric]; ok {
continue
}
sampleBuf = append(sampleBuf, metrics.Sample{Name: sumMetric})
sampleMap[sumMetric] = &sampleBuf[len(sampleBuf)-1]
}
var (
msMetrics memStatsMetrics
msDescriptions []metrics.Description
)
if !opt.DisableMemStatsLikeMetrics {
msMetrics = goRuntimeMemStats()
msDescriptions = bestEffortLookupRM(rmNamesForMemStatsMetrics)
// Check if metric was not exposed before and if not, add to sampleBuf.
for _, mdDesc := range msDescriptions {
if _, ok := sampleMap[mdDesc.Name]; ok {
continue
}
sampleBuf = append(sampleBuf, metrics.Sample{Name: mdDesc.Name})
sampleMap[mdDesc.Name] = &sampleBuf[len(sampleBuf)-1]
}
}
return &goCollector{
base: newBaseGoCollector(),
sampleBuf: sampleBuf,
sampleMap: sampleMap,
rmExposedMetrics: metricSet,
rmExactSumMapForHist: opt.RuntimeMetricSumForHist,
msMetrics: msMetrics,
msMetricsEnabled: !opt.DisableMemStatsLikeMetrics,
}
}
func attachOriginalName(desc, origName string) string {
return fmt.Sprintf("%s Sourced from %s.", desc, origName)
}
// Describe returns all descriptions of the collector.
func (c *goCollector) Describe(ch chan<- *Desc) {
c.base.Describe(ch)
for _, i := range c.msMetrics {
ch <- i.desc
}
for _, m := range c.rmExposedMetrics {
ch <- m.Desc()
}
}
// Collect returns the current state of all metrics of the collector.
func (c *goCollector) Collect(ch chan<- Metric) {
// Collect base non-memory metrics.
c.base.Collect(ch)
if len(c.sampleBuf) == 0 {
return
}
// Collect must be thread-safe, so prevent concurrent use of
// sampleBuf elements. Just read into sampleBuf but write all the data
// we get into our Metrics or MemStats.
//
// This lock also ensures that the Metrics we send out are all from
// the same updates, ensuring their mutual consistency insofar as
// is guaranteed by the runtime/metrics package.
//
// N.B. This locking is heavy-handed, but Collect is expected to be called
// relatively infrequently. Also the core operation here, metrics.Read,
// is fast (O(tens of microseconds)) so contention should certainly be
// low, though channel operations and any allocations may add to that.
c.mu.Lock()
defer c.mu.Unlock()
// Populate runtime/metrics sample buffer.
metrics.Read(c.sampleBuf)
// Collect all our runtime/metrics user chose to expose from sampleBuf (if any).
for i, metric := range c.rmExposedMetrics {
// We created samples for exposed metrics first in order, so indexes match.
sample := c.sampleBuf[i]
// N.B. switch on concrete type because it's significantly more efficient
// than checking for the Counter and Gauge interface implementations. In
// this case, we control all the types here.
switch m := metric.(type) {
case *counter:
// Guard against decreases. This should never happen, but a failure
// to do so will result in a panic, which is a harsh consequence for
// a metrics collection bug.
v0, v1 := m.get(), unwrapScalarRMValue(sample.Value)
if v1 > v0 {
m.Add(unwrapScalarRMValue(sample.Value) - m.get())
}
m.Collect(ch)
case *gauge:
m.Set(unwrapScalarRMValue(sample.Value))
m.Collect(ch)
case *batchHistogram:
m.update(sample.Value.Float64Histogram(), c.exactSumFor(sample.Name))
m.Collect(ch)
default:
panic("unexpected metric type")
}
}
if c.msMetricsEnabled {
// ms is a dummy MemStats that we populate ourselves so that we can
// populate the old metrics from it if goMemStatsCollection is enabled.
var ms runtime.MemStats
memStatsFromRM(&ms, c.sampleMap)
for _, i := range c.msMetrics {
ch <- MustNewConstMetric(i.desc, i.valType, i.eval(&ms))
}
}
}
// unwrapScalarRMValue unwraps a runtime/metrics value that is assumed
// to be scalar and returns the equivalent float64 value. Panics if the
// value is not scalar.
func unwrapScalarRMValue(v metrics.Value) float64 {
switch v.Kind() {
case metrics.KindUint64:
return float64(v.Uint64())
case metrics.KindFloat64:
return v.Float64()
case metrics.KindBad:
// Unsupported metric.
//
// This should never happen because we always populate our metric
// set from the runtime/metrics package.
panic("unexpected bad kind metric")
default:
// Unsupported metric kind.
//
// This should never happen because we check for this during initialization
// and flag and filter metrics whose kinds we don't understand.
panic(fmt.Sprintf("unexpected unsupported metric: %v", v.Kind()))
}
}
// exactSumFor takes a runtime/metrics metric name (that is assumed to
// be of kind KindFloat64Histogram) and returns its exact sum and whether
// its exact sum exists.
//
// The runtime/metrics API for histograms doesn't currently expose exact
// sums, but some of the other metrics are in fact exact sums of histograms.
func (c *goCollector) exactSumFor(rmName string) float64 {
sumName, ok := c.rmExactSumMapForHist[rmName]
if !ok {
return 0
}
s, ok := c.sampleMap[sumName]
if !ok {
return 0
}
return unwrapScalarRMValue(s.Value)
}
func memStatsFromRM(ms *runtime.MemStats, rm map[string]*metrics.Sample) {
lookupOrZero := func(name string) uint64 {
if s, ok := rm[name]; ok {
return s.Value.Uint64()
}
return 0
}
// Currently, MemStats adds tiny alloc count to both Mallocs AND Frees.
// The reason for this is because MemStats couldn't be extended at the time
// but there was a desire to have Mallocs at least be a little more representative,
// while having Mallocs - Frees still represent a live object count.
// Unfortunately, MemStats doesn't actually export a large allocation count,
// so it's impossible to pull this number out directly.
tinyAllocs := lookupOrZero(goGCHeapTinyAllocsObjects)
ms.Mallocs = lookupOrZero(goGCHeapAllocsObjects) + tinyAllocs
ms.Frees = lookupOrZero(goGCHeapFreesObjects) + tinyAllocs
ms.TotalAlloc = lookupOrZero(goGCHeapAllocsBytes)
ms.Sys = lookupOrZero(goMemoryClassesTotalBytes)
ms.Lookups = 0 // Already always zero.
ms.HeapAlloc = lookupOrZero(goMemoryClassesHeapObjectsBytes)
ms.Alloc = ms.HeapAlloc
ms.HeapInuse = ms.HeapAlloc + lookupOrZero(goMemoryClassesHeapUnusedBytes)
ms.HeapReleased = lookupOrZero(goMemoryClassesHeapReleasedBytes)
ms.HeapIdle = ms.HeapReleased + lookupOrZero(goMemoryClassesHeapFreeBytes)
ms.HeapSys = ms.HeapInuse + ms.HeapIdle
ms.HeapObjects = lookupOrZero(goGCHeapObjects)
ms.StackInuse = lookupOrZero(goMemoryClassesHeapStacksBytes)
ms.StackSys = ms.StackInuse + lookupOrZero(goMemoryClassesOSStacksBytes)
ms.MSpanInuse = lookupOrZero(goMemoryClassesMetadataMSpanInuseBytes)
ms.MSpanSys = ms.MSpanInuse + lookupOrZero(goMemoryClassesMetadataMSPanFreeBytes)
ms.MCacheInuse = lookupOrZero(goMemoryClassesMetadataMCacheInuseBytes)
ms.MCacheSys = ms.MCacheInuse + lookupOrZero(goMemoryClassesMetadataMCacheFreeBytes)
ms.BuckHashSys = lookupOrZero(goMemoryClassesProfilingBucketsBytes)
ms.GCSys = lookupOrZero(goMemoryClassesMetadataOtherBytes)
ms.OtherSys = lookupOrZero(goMemoryClassesOtherBytes)
ms.NextGC = lookupOrZero(goGCHeapGoalBytes)
// N.B. GCCPUFraction is intentionally omitted. This metric is not useful,
// and often misleading due to the fact that it's an average over the lifetime
// of the process.
// See https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034
// for more details.
ms.GCCPUFraction = 0
}
// batchHistogram is a mutable histogram that is updated
// in batches.
type batchHistogram struct {
selfCollector
// Static fields updated only once.
desc *Desc
hasSum bool
// Because this histogram operates in batches, it just uses a
// single mutex for everything. updates are always serialized
// but Write calls may operate concurrently with updates.
// Contention between these two sources should be rare.
mu sync.Mutex
buckets []float64 // Inclusive lower bounds, like runtime/metrics.
counts []uint64
sum float64 // Used if hasSum is true.
}
// newBatchHistogram creates a new batch histogram value with the given
// Desc, buckets, and whether or not it has an exact sum available.
//
// buckets must always be from the runtime/metrics package, following
// the same conventions.
func newBatchHistogram(desc *Desc, buckets []float64, hasSum bool) *batchHistogram {
// We need to remove -Inf values. runtime/metrics keeps them around.
// But -Inf bucket should not be allowed for prometheus histograms.
if buckets[0] == math.Inf(-1) {
buckets = buckets[1:]
}
h := &batchHistogram{
desc: desc,
buckets: buckets,
// Because buckets follows runtime/metrics conventions, there's
// 1 more value in the buckets list than there are buckets represented,
// because in runtime/metrics, the bucket values represent *boundaries*,
// and non-Inf boundaries are inclusive lower bounds for that bucket.
counts: make([]uint64, len(buckets)-1),
hasSum: hasSum,
}
h.init(h)
return h
}
// update updates the batchHistogram from a runtime/metrics histogram.
//
// sum must be provided if the batchHistogram was created to have an exact sum.
// h.buckets must be a strict subset of his.Buckets.
func (h *batchHistogram) update(his *metrics.Float64Histogram, sum float64) {
counts, buckets := his.Counts, his.Buckets
h.mu.Lock()
defer h.mu.Unlock()
// Clear buckets.
for i := range h.counts {
h.counts[i] = 0
}
// Copy and reduce buckets.
var j int
for i, count := range counts {
h.counts[j] += count
if buckets[i+1] == h.buckets[j+1] {
j++
}
}
if h.hasSum {
h.sum = sum
}
}
func (h *batchHistogram) Desc() *Desc {
return h.desc
}
func (h *batchHistogram) Write(out *dto.Metric) error {
h.mu.Lock()
defer h.mu.Unlock()
sum := float64(0)
if h.hasSum {
sum = h.sum
}
dtoBuckets := make([]*dto.Bucket, 0, len(h.counts))
totalCount := uint64(0)
for i, count := range h.counts {
totalCount += count
if !h.hasSum {
if count != 0 {
// N.B. This computed sum is an underestimate.
sum += h.buckets[i] * float64(count)
}
}
// Skip the +Inf bucket, but only for the bucket list.
// It must still count for sum and totalCount.
if math.IsInf(h.buckets[i+1], 1) {
break
}
// Float64Histogram's upper bound is exclusive, so make it inclusive
// by obtaining the next float64 value down, in order.
upperBound := math.Nextafter(h.buckets[i+1], h.buckets[i])
dtoBuckets = append(dtoBuckets, &dto.Bucket{
CumulativeCount: proto.Uint64(totalCount),
UpperBound: proto.Float64(upperBound),
})
}
out.Histogram = &dto.Histogram{
Bucket: dtoBuckets,
SampleCount: proto.Uint64(totalCount),
SampleSum: proto.Float64(sum),
}
return nil
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,60 @@
// Copyright (c) 2015 Björn Rabenstein
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
//
// The code in this package is copy/paste to avoid a dependency. Hence this file
// carries the copyright of the original repo.
// https://github.com/beorn7/floats
package internal
import (
"math"
)
// minNormalFloat64 is the smallest positive normal value of type float64.
var minNormalFloat64 = math.Float64frombits(0x0010000000000000)
// AlmostEqualFloat64 returns true if a and b are equal within a relative error
// of epsilon. See http://floating-point-gui.de/errors/comparison/ for the
// details of the applied method.
func AlmostEqualFloat64(a, b, epsilon float64) bool {
if a == b {
return true
}
absA := math.Abs(a)
absB := math.Abs(b)
diff := math.Abs(a - b)
if a == 0 || b == 0 || absA+absB < minNormalFloat64 {
return diff < epsilon*minNormalFloat64
}
return diff/math.Min(absA+absB, math.MaxFloat64) < epsilon
}
// AlmostEqualFloat64s is the slice form of AlmostEqualFloat64.
func AlmostEqualFloat64s(a, b []float64, epsilon float64) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if !AlmostEqualFloat64(a[i], b[i], epsilon) {
return false
}
}
return true
}

View File

@ -0,0 +1,655 @@
// Copyright 2022 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// It provides tools to compare sequences of strings and generate textual diffs.
//
// Maintaining `GetUnifiedDiffString` here because original repository
// (https://github.com/pmezard/go-difflib) is no longer maintained.
package internal
import (
"bufio"
"bytes"
"fmt"
"io"
"strconv"
"strings"
)
func minInt(a, b int) int {
if a < b {
return a
}
return b
}
func maxInt(a, b int) int {
if a > b {
return a
}
return b
}
func calculateRatio(matches, length int) float64 {
if length > 0 {
return 2.0 * float64(matches) / float64(length)
}
return 1.0
}
type Match struct {
A int
B int
Size int
}
type OpCode struct {
Tag byte
I1 int
I2 int
J1 int
J2 int
}
// SequenceMatcher compares sequence of strings. The basic
// algorithm predates, and is a little fancier than, an algorithm
// published in the late 1980's by Ratcliff and Obershelp under the
// hyperbolic name "gestalt pattern matching". The basic idea is to find
// the longest contiguous matching subsequence that contains no "junk"
// elements (R-O doesn't address junk). The same idea is then applied
// recursively to the pieces of the sequences to the left and to the right
// of the matching subsequence. This does not yield minimal edit
// sequences, but does tend to yield matches that "look right" to people.
//
// SequenceMatcher tries to compute a "human-friendly diff" between two
// sequences. Unlike e.g. UNIX(tm) diff, the fundamental notion is the
// longest *contiguous* & junk-free matching subsequence. That's what
// catches peoples' eyes. The Windows(tm) windiff has another interesting
// notion, pairing up elements that appear uniquely in each sequence.
// That, and the method here, appear to yield more intuitive difference
// reports than does diff. This method appears to be the least vulnerable
// to syncing up on blocks of "junk lines", though (like blank lines in
// ordinary text files, or maybe "<P>" lines in HTML files). That may be
// because this is the only method of the 3 that has a *concept* of
// "junk" <wink>.
//
// Timing: Basic R-O is cubic time worst case and quadratic time expected
// case. SequenceMatcher is quadratic time for the worst case and has
// expected-case behavior dependent in a complicated way on how many
// elements the sequences have in common; best case time is linear.
type SequenceMatcher struct {
a []string
b []string
b2j map[string][]int
IsJunk func(string) bool
autoJunk bool
bJunk map[string]struct{}
matchingBlocks []Match
fullBCount map[string]int
bPopular map[string]struct{}
opCodes []OpCode
}
func NewMatcher(a, b []string) *SequenceMatcher {
m := SequenceMatcher{autoJunk: true}
m.SetSeqs(a, b)
return &m
}
func NewMatcherWithJunk(a, b []string, autoJunk bool,
isJunk func(string) bool,
) *SequenceMatcher {
m := SequenceMatcher{IsJunk: isJunk, autoJunk: autoJunk}
m.SetSeqs(a, b)
return &m
}
// Set two sequences to be compared.
func (m *SequenceMatcher) SetSeqs(a, b []string) {
m.SetSeq1(a)
m.SetSeq2(b)
}
// Set the first sequence to be compared. The second sequence to be compared is
// not changed.
//
// SequenceMatcher computes and caches detailed information about the second
// sequence, so if you want to compare one sequence S against many sequences,
// use .SetSeq2(s) once and call .SetSeq1(x) repeatedly for each of the other
// sequences.
//
// See also SetSeqs() and SetSeq2().
func (m *SequenceMatcher) SetSeq1(a []string) {
if &a == &m.a {
return
}
m.a = a
m.matchingBlocks = nil
m.opCodes = nil
}
// Set the second sequence to be compared. The first sequence to be compared is
// not changed.
func (m *SequenceMatcher) SetSeq2(b []string) {
if &b == &m.b {
return
}
m.b = b
m.matchingBlocks = nil
m.opCodes = nil
m.fullBCount = nil
m.chainB()
}
func (m *SequenceMatcher) chainB() {
// Populate line -> index mapping
b2j := map[string][]int{}
for i, s := range m.b {
indices := b2j[s]
indices = append(indices, i)
b2j[s] = indices
}
// Purge junk elements
m.bJunk = map[string]struct{}{}
if m.IsJunk != nil {
junk := m.bJunk
for s := range b2j {
if m.IsJunk(s) {
junk[s] = struct{}{}
}
}
for s := range junk {
delete(b2j, s)
}
}
// Purge remaining popular elements
popular := map[string]struct{}{}
n := len(m.b)
if m.autoJunk && n >= 200 {
ntest := n/100 + 1
for s, indices := range b2j {
if len(indices) > ntest {
popular[s] = struct{}{}
}
}
for s := range popular {
delete(b2j, s)
}
}
m.bPopular = popular
m.b2j = b2j
}
func (m *SequenceMatcher) isBJunk(s string) bool {
_, ok := m.bJunk[s]
return ok
}
// Find longest matching block in a[alo:ahi] and b[blo:bhi].
//
// If IsJunk is not defined:
//
// Return (i,j,k) such that a[i:i+k] is equal to b[j:j+k], where
//
// alo <= i <= i+k <= ahi
// blo <= j <= j+k <= bhi
//
// and for all (i',j',k') meeting those conditions,
//
// k >= k'
// i <= i'
// and if i == i', j <= j'
//
// In other words, of all maximal matching blocks, return one that
// starts earliest in a, and of all those maximal matching blocks that
// start earliest in a, return the one that starts earliest in b.
//
// If IsJunk is defined, first the longest matching block is
// determined as above, but with the additional restriction that no
// junk element appears in the block. Then that block is extended as
// far as possible by matching (only) junk elements on both sides. So
// the resulting block never matches on junk except as identical junk
// happens to be adjacent to an "interesting" match.
//
// If no blocks match, return (alo, blo, 0).
func (m *SequenceMatcher) findLongestMatch(alo, ahi, blo, bhi int) Match {
// CAUTION: stripping common prefix or suffix would be incorrect.
// E.g.,
// ab
// acab
// Longest matching block is "ab", but if common prefix is
// stripped, it's "a" (tied with "b"). UNIX(tm) diff does so
// strip, so ends up claiming that ab is changed to acab by
// inserting "ca" in the middle. That's minimal but unintuitive:
// "it's obvious" that someone inserted "ac" at the front.
// Windiff ends up at the same place as diff, but by pairing up
// the unique 'b's and then matching the first two 'a's.
besti, bestj, bestsize := alo, blo, 0
// find longest junk-free match
// during an iteration of the loop, j2len[j] = length of longest
// junk-free match ending with a[i-1] and b[j]
j2len := map[int]int{}
for i := alo; i != ahi; i++ {
// look at all instances of a[i] in b; note that because
// b2j has no junk keys, the loop is skipped if a[i] is junk
newj2len := map[int]int{}
for _, j := range m.b2j[m.a[i]] {
// a[i] matches b[j]
if j < blo {
continue
}
if j >= bhi {
break
}
k := j2len[j-1] + 1
newj2len[j] = k
if k > bestsize {
besti, bestj, bestsize = i-k+1, j-k+1, k
}
}
j2len = newj2len
}
// Extend the best by non-junk elements on each end. In particular,
// "popular" non-junk elements aren't in b2j, which greatly speeds
// the inner loop above, but also means "the best" match so far
// doesn't contain any junk *or* popular non-junk elements.
for besti > alo && bestj > blo && !m.isBJunk(m.b[bestj-1]) &&
m.a[besti-1] == m.b[bestj-1] {
besti, bestj, bestsize = besti-1, bestj-1, bestsize+1
}
for besti+bestsize < ahi && bestj+bestsize < bhi &&
!m.isBJunk(m.b[bestj+bestsize]) &&
m.a[besti+bestsize] == m.b[bestj+bestsize] {
bestsize++
}
// Now that we have a wholly interesting match (albeit possibly
// empty!), we may as well suck up the matching junk on each
// side of it too. Can't think of a good reason not to, and it
// saves post-processing the (possibly considerable) expense of
// figuring out what to do with it. In the case of an empty
// interesting match, this is clearly the right thing to do,
// because no other kind of match is possible in the regions.
for besti > alo && bestj > blo && m.isBJunk(m.b[bestj-1]) &&
m.a[besti-1] == m.b[bestj-1] {
besti, bestj, bestsize = besti-1, bestj-1, bestsize+1
}
for besti+bestsize < ahi && bestj+bestsize < bhi &&
m.isBJunk(m.b[bestj+bestsize]) &&
m.a[besti+bestsize] == m.b[bestj+bestsize] {
bestsize++
}
return Match{A: besti, B: bestj, Size: bestsize}
}
// Return list of triples describing matching subsequences.
//
// Each triple is of the form (i, j, n), and means that
// a[i:i+n] == b[j:j+n]. The triples are monotonically increasing in
// i and in j. It's also guaranteed that if (i, j, n) and (i', j', n') are
// adjacent triples in the list, and the second is not the last triple in the
// list, then i+n != i' or j+n != j'. IOW, adjacent triples never describe
// adjacent equal blocks.
//
// The last triple is a dummy, (len(a), len(b), 0), and is the only
// triple with n==0.
func (m *SequenceMatcher) GetMatchingBlocks() []Match {
if m.matchingBlocks != nil {
return m.matchingBlocks
}
var matchBlocks func(alo, ahi, blo, bhi int, matched []Match) []Match
matchBlocks = func(alo, ahi, blo, bhi int, matched []Match) []Match {
match := m.findLongestMatch(alo, ahi, blo, bhi)
i, j, k := match.A, match.B, match.Size
if match.Size > 0 {
if alo < i && blo < j {
matched = matchBlocks(alo, i, blo, j, matched)
}
matched = append(matched, match)
if i+k < ahi && j+k < bhi {
matched = matchBlocks(i+k, ahi, j+k, bhi, matched)
}
}
return matched
}
matched := matchBlocks(0, len(m.a), 0, len(m.b), nil)
// It's possible that we have adjacent equal blocks in the
// matching_blocks list now.
nonAdjacent := []Match{}
i1, j1, k1 := 0, 0, 0
for _, b := range matched {
// Is this block adjacent to i1, j1, k1?
i2, j2, k2 := b.A, b.B, b.Size
if i1+k1 == i2 && j1+k1 == j2 {
// Yes, so collapse them -- this just increases the length of
// the first block by the length of the second, and the first
// block so lengthened remains the block to compare against.
k1 += k2
} else {
// Not adjacent. Remember the first block (k1==0 means it's
// the dummy we started with), and make the second block the
// new block to compare against.
if k1 > 0 {
nonAdjacent = append(nonAdjacent, Match{i1, j1, k1})
}
i1, j1, k1 = i2, j2, k2
}
}
if k1 > 0 {
nonAdjacent = append(nonAdjacent, Match{i1, j1, k1})
}
nonAdjacent = append(nonAdjacent, Match{len(m.a), len(m.b), 0})
m.matchingBlocks = nonAdjacent
return m.matchingBlocks
}
// Return list of 5-tuples describing how to turn a into b.
//
// Each tuple is of the form (tag, i1, i2, j1, j2). The first tuple
// has i1 == j1 == 0, and remaining tuples have i1 == the i2 from the
// tuple preceding it, and likewise for j1 == the previous j2.
//
// The tags are characters, with these meanings:
//
// 'r' (replace): a[i1:i2] should be replaced by b[j1:j2]
//
// 'd' (delete): a[i1:i2] should be deleted, j1==j2 in this case.
//
// 'i' (insert): b[j1:j2] should be inserted at a[i1:i1], i1==i2 in this case.
//
// 'e' (equal): a[i1:i2] == b[j1:j2]
func (m *SequenceMatcher) GetOpCodes() []OpCode {
if m.opCodes != nil {
return m.opCodes
}
i, j := 0, 0
matching := m.GetMatchingBlocks()
opCodes := make([]OpCode, 0, len(matching))
for _, m := range matching {
// invariant: we've pumped out correct diffs to change
// a[:i] into b[:j], and the next matching block is
// a[ai:ai+size] == b[bj:bj+size]. So we need to pump
// out a diff to change a[i:ai] into b[j:bj], pump out
// the matching block, and move (i,j) beyond the match
ai, bj, size := m.A, m.B, m.Size
tag := byte(0)
if i < ai && j < bj {
tag = 'r'
} else if i < ai {
tag = 'd'
} else if j < bj {
tag = 'i'
}
if tag > 0 {
opCodes = append(opCodes, OpCode{tag, i, ai, j, bj})
}
i, j = ai+size, bj+size
// the list of matching blocks is terminated by a
// sentinel with size 0
if size > 0 {
opCodes = append(opCodes, OpCode{'e', ai, i, bj, j})
}
}
m.opCodes = opCodes
return m.opCodes
}
// Isolate change clusters by eliminating ranges with no changes.
//
// Return a generator of groups with up to n lines of context.
// Each group is in the same format as returned by GetOpCodes().
func (m *SequenceMatcher) GetGroupedOpCodes(n int) [][]OpCode {
if n < 0 {
n = 3
}
codes := m.GetOpCodes()
if len(codes) == 0 {
codes = []OpCode{{'e', 0, 1, 0, 1}}
}
// Fixup leading and trailing groups if they show no changes.
if codes[0].Tag == 'e' {
c := codes[0]
i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2
codes[0] = OpCode{c.Tag, maxInt(i1, i2-n), i2, maxInt(j1, j2-n), j2}
}
if codes[len(codes)-1].Tag == 'e' {
c := codes[len(codes)-1]
i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2
codes[len(codes)-1] = OpCode{c.Tag, i1, minInt(i2, i1+n), j1, minInt(j2, j1+n)}
}
nn := n + n
groups := [][]OpCode{}
group := []OpCode{}
for _, c := range codes {
i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2
// End the current group and start a new one whenever
// there is a large range with no changes.
if c.Tag == 'e' && i2-i1 > nn {
group = append(group, OpCode{
c.Tag, i1, minInt(i2, i1+n),
j1, minInt(j2, j1+n),
})
groups = append(groups, group)
group = []OpCode{}
i1, j1 = maxInt(i1, i2-n), maxInt(j1, j2-n)
}
group = append(group, OpCode{c.Tag, i1, i2, j1, j2})
}
if len(group) > 0 && (len(group) != 1 || group[0].Tag != 'e') {
groups = append(groups, group)
}
return groups
}
// Return a measure of the sequences' similarity (float in [0,1]).
//
// Where T is the total number of elements in both sequences, and
// M is the number of matches, this is 2.0*M / T.
// Note that this is 1 if the sequences are identical, and 0 if
// they have nothing in common.
//
// .Ratio() is expensive to compute if you haven't already computed
// .GetMatchingBlocks() or .GetOpCodes(), in which case you may
// want to try .QuickRatio() or .RealQuickRation() first to get an
// upper bound.
func (m *SequenceMatcher) Ratio() float64 {
matches := 0
for _, m := range m.GetMatchingBlocks() {
matches += m.Size
}
return calculateRatio(matches, len(m.a)+len(m.b))
}
// Return an upper bound on ratio() relatively quickly.
//
// This isn't defined beyond that it is an upper bound on .Ratio(), and
// is faster to compute.
func (m *SequenceMatcher) QuickRatio() float64 {
// viewing a and b as multisets, set matches to the cardinality
// of their intersection; this counts the number of matches
// without regard to order, so is clearly an upper bound
if m.fullBCount == nil {
m.fullBCount = map[string]int{}
for _, s := range m.b {
m.fullBCount[s]++
}
}
// avail[x] is the number of times x appears in 'b' less the
// number of times we've seen it in 'a' so far ... kinda
avail := map[string]int{}
matches := 0
for _, s := range m.a {
n, ok := avail[s]
if !ok {
n = m.fullBCount[s]
}
avail[s] = n - 1
if n > 0 {
matches++
}
}
return calculateRatio(matches, len(m.a)+len(m.b))
}
// Return an upper bound on ratio() very quickly.
//
// This isn't defined beyond that it is an upper bound on .Ratio(), and
// is faster to compute than either .Ratio() or .QuickRatio().
func (m *SequenceMatcher) RealQuickRatio() float64 {
la, lb := len(m.a), len(m.b)
return calculateRatio(minInt(la, lb), la+lb)
}
// Convert range to the "ed" format
func formatRangeUnified(start, stop int) string {
// Per the diff spec at http://www.unix.org/single_unix_specification/
beginning := start + 1 // lines start numbering with one
length := stop - start
if length == 1 {
return strconv.Itoa(beginning)
}
if length == 0 {
beginning-- // empty ranges begin at line just before the range
}
return fmt.Sprintf("%d,%d", beginning, length)
}
// Unified diff parameters
type UnifiedDiff struct {
A []string // First sequence lines
FromFile string // First file name
FromDate string // First file time
B []string // Second sequence lines
ToFile string // Second file name
ToDate string // Second file time
Eol string // Headers end of line, defaults to LF
Context int // Number of context lines
}
// Compare two sequences of lines; generate the delta as a unified diff.
//
// Unified diffs are a compact way of showing line changes and a few
// lines of context. The number of context lines is set by 'n' which
// defaults to three.
//
// By default, the diff control lines (those with ---, +++, or @@) are
// created with a trailing newline. This is helpful so that inputs
// created from file.readlines() result in diffs that are suitable for
// file.writelines() since both the inputs and outputs have trailing
// newlines.
//
// For inputs that do not have trailing newlines, set the lineterm
// argument to "" so that the output will be uniformly newline free.
//
// The unidiff format normally has a header for filenames and modification
// times. Any or all of these may be specified using strings for
// 'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'.
// The modification times are normally expressed in the ISO 8601 format.
func WriteUnifiedDiff(writer io.Writer, diff UnifiedDiff) error {
buf := bufio.NewWriter(writer)
defer buf.Flush()
wf := func(format string, args ...any) error {
_, err := fmt.Fprintf(buf, format, args...)
return err
}
ws := func(s string) error {
_, err := buf.WriteString(s)
return err
}
if len(diff.Eol) == 0 {
diff.Eol = "\n"
}
started := false
m := NewMatcher(diff.A, diff.B)
for _, g := range m.GetGroupedOpCodes(diff.Context) {
if !started {
started = true
fromDate := ""
if len(diff.FromDate) > 0 {
fromDate = "\t" + diff.FromDate
}
toDate := ""
if len(diff.ToDate) > 0 {
toDate = "\t" + diff.ToDate
}
if diff.FromFile != "" || diff.ToFile != "" {
err := wf("--- %s%s%s", diff.FromFile, fromDate, diff.Eol)
if err != nil {
return err
}
err = wf("+++ %s%s%s", diff.ToFile, toDate, diff.Eol)
if err != nil {
return err
}
}
}
first, last := g[0], g[len(g)-1]
range1 := formatRangeUnified(first.I1, last.I2)
range2 := formatRangeUnified(first.J1, last.J2)
if err := wf("@@ -%s +%s @@%s", range1, range2, diff.Eol); err != nil {
return err
}
for _, c := range g {
i1, i2, j1, j2 := c.I1, c.I2, c.J1, c.J2
if c.Tag == 'e' {
for _, line := range diff.A[i1:i2] {
if err := ws(" " + line); err != nil {
return err
}
}
continue
}
if c.Tag == 'r' || c.Tag == 'd' {
for _, line := range diff.A[i1:i2] {
if err := ws("-" + line); err != nil {
return err
}
}
}
if c.Tag == 'r' || c.Tag == 'i' {
for _, line := range diff.B[j1:j2] {
if err := ws("+" + line); err != nil {
return err
}
}
}
}
}
return nil
}
// Like WriteUnifiedDiff but returns the diff a string.
func GetUnifiedDiffString(diff UnifiedDiff) (string, error) {
w := &bytes.Buffer{}
err := WriteUnifiedDiff(w, diff)
return w.String(), err
}
// Split a string on "\n" while preserving them. The output can be used
// as input for UnifiedDiff and ContextDiff structures.
func SplitLines(s string) []string {
lines := strings.SplitAfter(s, "\n")
lines[len(lines)-1] += "\n"
return lines
}

View File

@ -0,0 +1,34 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package internal
import "regexp"
type GoCollectorRule struct {
Matcher *regexp.Regexp
Deny bool
}
// GoCollectorOptions should not be used be directly by anything, except `collectors` package.
// Use it via collectors package instead. See issue
// https://github.com/prometheus/client_golang/issues/1030.
//
// This is internal, so external users only can use it via `collector.WithGoCollector*` methods
type GoCollectorOptions struct {
DisableMemStatsLikeMetrics bool
RuntimeMetricSumForHist map[string]string
RuntimeMetricRules []GoCollectorRule
}
var GoCollectorDefaultRuntimeMetrics = regexp.MustCompile(`/gc/gogc:percent|/gc/gomemlimit:bytes|/sched/gomaxprocs:threads`)

View File

@ -0,0 +1,143 @@
// Copyright 2021 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build go1.17
// +build go1.17
package internal
import (
"math"
"path"
"runtime/metrics"
"strings"
"github.com/prometheus/common/model"
)
// RuntimeMetricsToProm produces a Prometheus metric name from a runtime/metrics
// metric description and validates whether the metric is suitable for integration
// with Prometheus.
//
// Returns false if a name could not be produced, or if Prometheus does not understand
// the runtime/metrics Kind.
//
// Note that the main reason a name couldn't be produced is if the runtime/metrics
// package exports a name with characters outside the valid Prometheus metric name
// character set. This is theoretically possible, but should never happen in practice.
// Still, don't rely on it.
func RuntimeMetricsToProm(d *metrics.Description) (string, string, string, bool) {
namespace := "go"
comp := strings.SplitN(d.Name, ":", 2)
key := comp[0]
unit := comp[1]
// The last path element in the key is the name,
// the rest is the subsystem.
subsystem := path.Dir(key[1:] /* remove leading / */)
name := path.Base(key)
// subsystem is translated by replacing all / and - with _.
subsystem = strings.ReplaceAll(subsystem, "/", "_")
subsystem = strings.ReplaceAll(subsystem, "-", "_")
// unit is translated assuming that the unit contains no
// non-ASCII characters.
unit = strings.ReplaceAll(unit, "-", "_")
unit = strings.ReplaceAll(unit, "*", "_")
unit = strings.ReplaceAll(unit, "/", "_per_")
// name has - replaced with _ and is concatenated with the unit and
// other data.
name = strings.ReplaceAll(name, "-", "_")
name += "_" + unit
if d.Cumulative && d.Kind != metrics.KindFloat64Histogram {
name += "_total"
}
// Our current conversion moves to legacy naming, so use legacy validation.
valid := model.LegacyValidation.IsValidMetricName(namespace + "_" + subsystem + "_" + name)
switch d.Kind {
case metrics.KindUint64:
case metrics.KindFloat64:
case metrics.KindFloat64Histogram:
default:
valid = false
}
return namespace, subsystem, name, valid
}
// RuntimeMetricsBucketsForUnit takes a set of buckets obtained for a runtime/metrics histogram
// type (so, lower-bound inclusive) and a unit from a runtime/metrics name, and produces
// a reduced set of buckets. This function always removes any -Inf bucket as it's represented
// as the bottom-most upper-bound inclusive bucket in Prometheus.
func RuntimeMetricsBucketsForUnit(buckets []float64, unit string) []float64 {
switch unit {
case "bytes":
// Re-bucket as powers of 2.
return reBucketExp(buckets, 2)
case "seconds":
// Re-bucket as powers of 10 and then merge all buckets greater
// than 1 second into the +Inf bucket.
b := reBucketExp(buckets, 10)
for i := range b {
if b[i] <= 1 {
continue
}
b[i] = math.Inf(1)
b = b[:i+1]
break
}
return b
}
return buckets
}
// reBucketExp takes a list of bucket boundaries (lower bound inclusive) and
// downsamples the buckets to those a multiple of base apart. The end result
// is a roughly exponential (in many cases, perfectly exponential) bucketing
// scheme.
func reBucketExp(buckets []float64, base float64) []float64 {
bucket := buckets[0]
var newBuckets []float64
// We may see a -Inf here, in which case, add it and skip it
// since we risk producing NaNs otherwise.
//
// We need to preserve -Inf values to maintain runtime/metrics
// conventions. We'll strip it out later.
if bucket == math.Inf(-1) {
newBuckets = append(newBuckets, bucket)
buckets = buckets[1:]
bucket = buckets[0]
}
// From now on, bucket should always have a non-Inf value because
// Infs are only ever at the ends of the bucket lists, so
// arithmetic operations on it are non-NaN.
for i := 1; i < len(buckets); i++ {
if bucket >= 0 && buckets[i] < bucket*base {
// The next bucket we want to include is at least bucket*base.
continue
} else if bucket < 0 && buckets[i] < bucket/base {
// In this case the bucket we're targeting is negative, and since
// we're ascending through buckets here, we need to divide to get
// closer to zero exponentially.
continue
}
// The +Inf bucket will always be the last one, and we'll always
// end up including it here because bucket
newBuckets = append(newBuckets, bucket)
bucket = buckets[i]
}
return append(newBuckets, bucket)
}

View File

@ -0,0 +1,101 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package internal
import (
"sort"
dto "github.com/prometheus/client_model/go"
)
// LabelPairSorter implements sort.Interface. It is used to sort a slice of
// dto.LabelPair pointers.
type LabelPairSorter []*dto.LabelPair
func (s LabelPairSorter) Len() int {
return len(s)
}
func (s LabelPairSorter) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s LabelPairSorter) Less(i, j int) bool {
return s[i].GetName() < s[j].GetName()
}
// MetricSorter is a sortable slice of *dto.Metric.
type MetricSorter []*dto.Metric
func (s MetricSorter) Len() int {
return len(s)
}
func (s MetricSorter) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}
func (s MetricSorter) Less(i, j int) bool {
if len(s[i].Label) != len(s[j].Label) {
// This should not happen. The metrics are
// inconsistent. However, we have to deal with the fact, as
// people might use custom collectors or metric family injection
// to create inconsistent metrics. So let's simply compare the
// number of labels in this case. That will still yield
// reproducible sorting.
return len(s[i].Label) < len(s[j].Label)
}
for n, lp := range s[i].Label {
vi := lp.GetValue()
vj := s[j].Label[n].GetValue()
if vi != vj {
return vi < vj
}
}
// We should never arrive here. Multiple metrics with the same
// label set in the same scrape will lead to undefined ingestion
// behavior. However, as above, we have to provide stable sorting
// here, even for inconsistent metrics. So sort equal metrics
// by their timestamp, with missing timestamps (implying "now")
// coming last.
if s[i].TimestampMs == nil {
return false
}
if s[j].TimestampMs == nil {
return true
}
return s[i].GetTimestampMs() < s[j].GetTimestampMs()
}
// NormalizeMetricFamilies returns a MetricFamily slice with empty
// MetricFamilies pruned and the remaining MetricFamilies sorted by name within
// the slice, with the contained Metrics sorted within each MetricFamily.
func NormalizeMetricFamilies(metricFamiliesByName map[string]*dto.MetricFamily) []*dto.MetricFamily {
for _, mf := range metricFamiliesByName {
sort.Sort(MetricSorter(mf.Metric))
}
names := make([]string, 0, len(metricFamiliesByName))
for name, mf := range metricFamiliesByName {
if len(mf.Metric) > 0 {
names = append(names, name)
}
}
sort.Strings(names)
result := make([]*dto.MetricFamily, 0, len(names))
for _, name := range names {
result = append(result, metricFamiliesByName[name])
}
return result
}

View File

@ -0,0 +1,188 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"errors"
"fmt"
"strings"
"unicode/utf8"
"github.com/prometheus/common/model"
)
// Labels represents a collection of label name -> value mappings. This type is
// commonly used with the With(Labels) and GetMetricWith(Labels) methods of
// metric vector Collectors, e.g.:
//
// myVec.With(Labels{"code": "404", "method": "GET"}).Add(42)
//
// The other use-case is the specification of constant label pairs in Opts or to
// create a Desc.
type Labels map[string]string
// LabelConstraint normalizes label values.
type LabelConstraint func(string) string
// ConstrainedLabels represents a label name and its constrain function
// to normalize label values. This type is commonly used when constructing
// metric vector Collectors.
type ConstrainedLabel struct {
Name string
Constraint LabelConstraint
}
// ConstrainableLabels is an interface that allows creating of labels that can
// be optionally constrained.
//
// prometheus.V2().NewCounterVec(CounterVecOpts{
// CounterOpts: {...}, // Usual CounterOpts fields
// VariableLabels: []ConstrainedLabels{
// {Name: "A"},
// {Name: "B", Constraint: func(v string) string { ... }},
// },
// })
type ConstrainableLabels interface {
compile() *compiledLabels
labelNames() []string
}
// ConstrainedLabels represents a collection of label name -> constrain function
// to normalize label values. This type is commonly used when constructing
// metric vector Collectors.
type ConstrainedLabels []ConstrainedLabel
func (cls ConstrainedLabels) compile() *compiledLabels {
compiled := &compiledLabels{
names: make([]string, len(cls)),
labelConstraints: map[string]LabelConstraint{},
}
for i, label := range cls {
compiled.names[i] = label.Name
if label.Constraint != nil {
compiled.labelConstraints[label.Name] = label.Constraint
}
}
return compiled
}
func (cls ConstrainedLabels) labelNames() []string {
names := make([]string, len(cls))
for i, label := range cls {
names[i] = label.Name
}
return names
}
// UnconstrainedLabels represents collection of label without any constraint on
// their value. Thus, it is simply a collection of label names.
//
// UnconstrainedLabels([]string{ "A", "B" })
//
// is equivalent to
//
// ConstrainedLabels {
// { Name: "A" },
// { Name: "B" },
// }
type UnconstrainedLabels []string
func (uls UnconstrainedLabels) compile() *compiledLabels {
return &compiledLabels{
names: uls,
}
}
func (uls UnconstrainedLabels) labelNames() []string {
return uls
}
type compiledLabels struct {
names []string
labelConstraints map[string]LabelConstraint
}
func (cls *compiledLabels) compile() *compiledLabels {
return cls
}
func (cls *compiledLabels) labelNames() []string {
return cls.names
}
func (cls *compiledLabels) constrain(labelName, value string) string {
if fn, ok := cls.labelConstraints[labelName]; ok && fn != nil {
return fn(value)
}
return value
}
// reservedLabelPrefix is a prefix which is not legal in user-supplied
// label names.
const reservedLabelPrefix = "__"
var errInconsistentCardinality = errors.New("inconsistent label cardinality")
func makeInconsistentCardinalityError(fqName string, labels, labelValues []string) error {
return fmt.Errorf(
"%w: %q has %d variable labels named %q but %d values %q were provided",
errInconsistentCardinality, fqName,
len(labels), labels,
len(labelValues), labelValues,
)
}
func validateValuesInLabels(labels Labels, expectedNumberOfValues int) error {
if len(labels) != expectedNumberOfValues {
return fmt.Errorf(
"%w: expected %d label values but got %d in %#v",
errInconsistentCardinality, expectedNumberOfValues,
len(labels), labels,
)
}
for name, val := range labels {
if !utf8.ValidString(val) {
return fmt.Errorf("label %s: value %q is not valid UTF-8", name, val)
}
}
return nil
}
func validateLabelValues(vals []string, expectedNumberOfValues int) error {
if len(vals) != expectedNumberOfValues {
// The call below makes vals escape, copy them to avoid that.
vals := append([]string(nil), vals...)
return fmt.Errorf(
"%w: expected %d label values but got %d in %#v",
errInconsistentCardinality, expectedNumberOfValues,
len(vals), vals,
)
}
for _, val := range vals {
if !utf8.ValidString(val) {
return fmt.Errorf("label value %q is not valid UTF-8", val)
}
}
return nil
}
func checkLabelName(l string) bool {
return model.UTF8Validation.IsValidLabelName(l) && !strings.HasPrefix(l, reservedLabelPrefix)
}

View File

@ -0,0 +1,279 @@
// Copyright 2014 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"errors"
"math"
"sort"
"strings"
"time"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/model"
"google.golang.org/protobuf/proto"
)
var separatorByteSlice = []byte{model.SeparatorByte} // For convenient use with xxhash.
// A Metric models a single sample value with its meta data being exported to
// Prometheus. Implementations of Metric in this package are Gauge, Counter,
// Histogram, Summary, and Untyped.
type Metric interface {
// Desc returns the descriptor for the Metric. This method idempotently
// returns the same descriptor throughout the lifetime of the
// Metric. The returned descriptor is immutable by contract. A Metric
// unable to describe itself must return an invalid descriptor (created
// with NewInvalidDesc).
Desc() *Desc
// Write encodes the Metric into a "Metric" Protocol Buffer data
// transmission object.
//
// Metric implementations must observe concurrency safety as reads of
// this metric may occur at any time, and any blocking occurs at the
// expense of total performance of rendering all registered
// metrics. Ideally, Metric implementations should support concurrent
// readers.
//
// While populating dto.Metric, it is the responsibility of the
// implementation to ensure validity of the Metric protobuf (like valid
// UTF-8 strings or syntactically valid metric and label names). It is
// recommended to sort labels lexicographically. Callers of Write should
// still make sure of sorting if they depend on it.
Write(*dto.Metric) error
// TODO(beorn7): The original rationale of passing in a pre-allocated
// dto.Metric protobuf to save allocations has disappeared. The
// signature of this method should be changed to "Write() (*dto.Metric,
// error)".
}
// Opts bundles the options for creating most Metric types. Each metric
// implementation XXX has its own XXXOpts type, but in most cases, it is just
// an alias of this type (which might change when the requirement arises.)
//
// It is mandatory to set Name to a non-empty string. All other fields are
// optional and can safely be left at their zero value, although it is strongly
// encouraged to set a Help string.
type Opts struct {
// Namespace, Subsystem, and Name are components of the fully-qualified
// name of the Metric (created by joining these components with
// "_"). Only Name is mandatory, the others merely help structuring the
// name. Note that the fully-qualified name of the metric must be a
// valid Prometheus metric name.
Namespace string
Subsystem string
Name string
// Help provides information about this metric.
//
// Metrics with the same fully-qualified name must have the same Help
// string.
Help string
// Unit provides the unit of this metric as per https://prometheus.io/docs/specs/om
Unit string
// ConstLabels are used to attach fixed labels to this metric. Metrics
// with the same fully-qualified name must have the same label names in
// their ConstLabels.
//
// ConstLabels are only used rarely. In particular, do not use them to
// attach the same labels to all your metrics. Those use cases are
// better covered by target labels set by the scraping Prometheus
// server, or by one specific metric (e.g. a build_info or a
// machine_role metric). See also
// https://prometheus.io/docs/instrumenting/writing_exporters/#target-labels-not-static-scraped-labels
ConstLabels Labels
// now is for testing purposes, by default it's time.Now.
now func() time.Time
}
// BuildFQName joins the given three name components by "_". Empty name
// components are ignored. If the name parameter itself is empty, an empty
// string is returned, no matter what. Metric implementations included in this
// library use this function internally to generate the fully-qualified metric
// name from the name component in their Opts. Users of the library will only
// need this function if they implement their own Metric or instantiate a Desc
// (with NewDesc) directly.
func BuildFQName(namespace, subsystem, name string) string {
if name == "" {
return ""
}
sb := strings.Builder{}
sb.Grow(len(namespace) + len(subsystem) + len(name) + 2)
if namespace != "" {
sb.WriteString(namespace)
sb.WriteString("_")
}
if subsystem != "" {
sb.WriteString(subsystem)
sb.WriteString("_")
}
sb.WriteString(name)
return sb.String()
}
type invalidMetric struct {
desc *Desc
err error
}
// NewInvalidMetric returns a metric whose Write method always returns the
// provided error. It is useful if a Collector finds itself unable to collect
// a metric and wishes to report an error to the registry.
func NewInvalidMetric(desc *Desc, err error) Metric {
return &invalidMetric{desc, err}
}
func (m *invalidMetric) Desc() *Desc { return m.desc }
func (m *invalidMetric) Write(*dto.Metric) error { return m.err }
type timestampedMetric struct {
Metric
t time.Time
}
func (m timestampedMetric) Write(pb *dto.Metric) error {
e := m.Metric.Write(pb)
pb.TimestampMs = proto.Int64(m.t.Unix()*1000 + int64(m.t.Nanosecond()/1000000))
return e
}
// NewMetricWithTimestamp returns a new Metric wrapping the provided Metric in a
// way that it has an explicit timestamp set to the provided Time. This is only
// useful in rare cases as the timestamp of a Prometheus metric should usually
// be set by the Prometheus server during scraping. Exceptions include mirroring
// metrics with given timestamps from other metric
// sources.
//
// NewMetricWithTimestamp works best with MustNewConstMetric,
// MustNewConstHistogram, and MustNewConstSummary, see example.
//
// Currently, the exposition formats used by Prometheus are limited to
// millisecond resolution. Thus, the provided time will be rounded down to the
// next full millisecond value.
func NewMetricWithTimestamp(t time.Time, m Metric) Metric {
return timestampedMetric{Metric: m, t: t}
}
type withExemplarsMetric struct {
Metric
exemplars []*dto.Exemplar
}
func (m *withExemplarsMetric) Write(pb *dto.Metric) error {
if err := m.Metric.Write(pb); err != nil {
return err
}
switch {
case pb.Counter != nil:
pb.Counter.Exemplar = m.exemplars[len(m.exemplars)-1]
case pb.Histogram != nil:
h := pb.Histogram
for _, e := range m.exemplars {
if (h.GetZeroThreshold() != 0 || h.GetZeroCount() != 0 ||
len(h.PositiveSpan) != 0 || len(h.NegativeSpan) != 0) &&
e.GetTimestamp() != nil {
h.Exemplars = append(h.Exemplars, e)
if len(h.Bucket) == 0 {
// Don't proceed to classic buckets if there are none.
continue
}
}
// h.Bucket are sorted by UpperBound.
i := sort.Search(len(h.Bucket), func(i int) bool {
return h.Bucket[i].GetUpperBound() >= e.GetValue()
})
if i < len(h.Bucket) {
h.Bucket[i].Exemplar = e
} else {
// The +Inf bucket should be explicitly added if there is an exemplar for it, similar to non-const histogram logic in https://github.com/prometheus/client_golang/blob/main/prometheus/histogram.go#L357-L365.
b := &dto.Bucket{
CumulativeCount: proto.Uint64(h.GetSampleCount()),
UpperBound: proto.Float64(math.Inf(1)),
Exemplar: e,
}
h.Bucket = append(h.Bucket, b)
}
}
default:
// TODO(bwplotka): Implement Gauge?
return errors.New("cannot inject exemplar into Gauge, Summary or Untyped")
}
return nil
}
// Exemplar is easier to use, user-facing representation of *dto.Exemplar.
type Exemplar struct {
Value float64
Labels Labels
// Optional.
// Default value (time.Time{}) indicates its empty, which should be
// understood as time.Now() time at the moment of creation of metric.
Timestamp time.Time
}
// NewMetricWithExemplars returns a new Metric wrapping the provided Metric with given
// exemplars. Exemplars are validated.
//
// Only last applicable exemplar is injected from the list.
// For example for Counter it means last exemplar is injected.
// For Histogram, it means last applicable exemplar for each bucket is injected.
// For a Native Histogram, all valid exemplars are injected.
//
// NewMetricWithExemplars works best with MustNewConstMetric and
// MustNewConstHistogram, see example.
func NewMetricWithExemplars(m Metric, exemplars ...Exemplar) (Metric, error) {
if len(exemplars) == 0 {
return nil, errors.New("no exemplar was passed for NewMetricWithExemplars")
}
var (
now = time.Now()
exs = make([]*dto.Exemplar, len(exemplars))
err error
)
for i, e := range exemplars {
ts := e.Timestamp
if ts.IsZero() {
ts = now
}
exs[i], err = newExemplar(e.Value, ts, e.Labels)
if err != nil {
return nil, err
}
}
return &withExemplarsMetric{Metric: m, exemplars: exs}, nil
}
// MustNewMetricWithExemplars is a version of NewMetricWithExemplars that panics where
// NewMetricWithExemplars would have returned an error.
func MustNewMetricWithExemplars(m Metric, exemplars ...Exemplar) Metric {
ret, err := NewMetricWithExemplars(m, exemplars...)
if err != nil {
panic(err)
}
return ret
}

View File

@ -0,0 +1,25 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build !js || wasm
// +build !js wasm
package prometheus
import "runtime"
// getRuntimeNumThreads returns the number of open OS threads.
func getRuntimeNumThreads() float64 {
n, _ := runtime.ThreadCreateProfile(nil)
return float64(n)
}

View File

@ -0,0 +1,22 @@
// Copyright 2018 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//go:build js && !wasm
// +build js,!wasm
package prometheus
// getRuntimeNumThreads returns the number of open OS threads.
func getRuntimeNumThreads() float64 {
return 1
}

View File

@ -0,0 +1,64 @@
// Copyright 2017 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
// Observer is the interface that wraps the Observe method, which is used by
// Histogram and Summary to add observations.
type Observer interface {
Observe(float64)
}
// The ObserverFunc type is an adapter to allow the use of ordinary
// functions as Observers. If f is a function with the appropriate
// signature, ObserverFunc(f) is an Observer that calls f.
//
// This adapter is usually used in connection with the Timer type, and there are
// two general use cases:
//
// The most common one is to use a Gauge as the Observer for a Timer.
// See the "Gauge" Timer example.
//
// The more advanced use case is to create a function that dynamically decides
// which Observer to use for observing the duration. See the "Complex" Timer
// example.
type ObserverFunc func(float64)
// Observe calls f(value). It implements Observer.
func (f ObserverFunc) Observe(value float64) {
f(value)
}
// ObserverVec is an interface implemented by `HistogramVec` and `SummaryVec`.
type ObserverVec interface {
GetMetricWith(Labels) (Observer, error)
GetMetricWithLabelValues(lvs ...string) (Observer, error)
With(Labels) Observer
WithLabelValues(...string) Observer
CurryWith(Labels) (ObserverVec, error)
MustCurryWith(Labels) ObserverVec
Collector
}
// ExemplarObserver is implemented by Observers that offer the option of
// observing a value together with an exemplar. Its ObserveWithExemplar method
// works like the Observe method of an Observer but also replaces the currently
// saved exemplar (if any) with a new one, created from the provided value, the
// current time as timestamp, and the provided Labels. Empty Labels will lead to
// a valid (label-less) exemplar. But if Labels is nil, the current exemplar is
// left in place. ObserveWithExemplar panics if any of the provided labels are
// invalid or if the provided labels contain more than 128 runes in total.
type ExemplarObserver interface {
ObserveWithExemplar(value float64, exemplar Labels)
}

View File

@ -0,0 +1,180 @@
// Copyright 2015 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package prometheus
import (
"errors"
"fmt"
"os"
"strconv"
"strings"
)
type processCollector struct {
collectFn func(chan<- Metric)
describeFn func(chan<- *Desc)
pidFn func() (int, error)
reportErrors bool
cpuTotal *Desc
openFDs, maxFDs *Desc
vsize, maxVsize *Desc
rss *Desc
startTime *Desc
inBytes, outBytes *Desc
}
// ProcessCollectorOpts defines the behavior of a process metrics collector
// created with NewProcessCollector.
type ProcessCollectorOpts struct {
// PidFn returns the PID of the process the collector collects metrics
// for. It is called upon each collection. By default, the PID of the
// current process is used, as determined on construction time by
// calling os.Getpid().
PidFn func() (int, error)
// If non-empty, each of the collected metrics is prefixed by the
// provided string and an underscore ("_").
Namespace string
// If true, any error encountered during collection is reported as an
// invalid metric (see NewInvalidMetric). Otherwise, errors are ignored
// and the collected metrics will be incomplete. (Possibly, no metrics
// will be collected at all.) While that's usually not desired, it is
// appropriate for the common "mix-in" of process metrics, where process
// metrics are nice to have, but failing to collect them should not
// disrupt the collection of the remaining metrics.
ReportErrors bool
}
// NewProcessCollector is the obsolete version of collectors.NewProcessCollector.
// See there for documentation.
//
// Deprecated: Use collectors.NewProcessCollector instead.
func NewProcessCollector(opts ProcessCollectorOpts) Collector {
ns := ""
if len(opts.Namespace) > 0 {
ns = opts.Namespace + "_"
}
c := &processCollector{
reportErrors: opts.ReportErrors,
cpuTotal: NewDesc(
ns+"process_cpu_seconds_total",
"Total user and system CPU time spent in seconds.",
nil, nil,
),
openFDs: NewDesc(
ns+"process_open_fds",
"Number of open file descriptors.",
nil, nil,
),
maxFDs: NewDesc(
ns+"process_max_fds",
"Maximum number of open file descriptors.",
nil, nil,
),
vsize: NewDesc(
ns+"process_virtual_memory_bytes",
"Virtual memory size in bytes.",
nil, nil,
),
maxVsize: NewDesc(
ns+"process_virtual_memory_max_bytes",
"Maximum amount of virtual memory available in bytes.",
nil, nil,
),
rss: NewDesc(
ns+"process_resident_memory_bytes",
"Resident memory size in bytes.",
nil, nil,
),
startTime: NewDesc(
ns+"process_start_time_seconds",
"Start time of the process since unix epoch in seconds.",
nil, nil,
),
inBytes: NewDesc(
ns+"process_network_receive_bytes_total",
"Number of bytes received by the process over the network.",
nil, nil,
),
outBytes: NewDesc(
ns+"process_network_transmit_bytes_total",
"Number of bytes sent by the process over the network.",
nil, nil,
),
}
if opts.PidFn == nil {
c.pidFn = getPIDFn()
} else {
c.pidFn = opts.PidFn
}
// Set up process metric collection if supported by the runtime.
if canCollectProcess() {
c.collectFn = c.processCollect
c.describeFn = c.describe
} else {
c.collectFn = c.errorCollectFn
c.describeFn = c.errorDescribeFn
}
return c
}
func (c *processCollector) errorCollectFn(ch chan<- Metric) {
c.reportError(ch, nil, errors.New("process metrics not supported on this platform"))
}
func (c *processCollector) errorDescribeFn(ch chan<- *Desc) {
if c.reportErrors {
ch <- NewInvalidDesc(errors.New("process metrics not supported on this platform"))
}
}
// Collect returns the current state of all metrics of the collector.
func (c *processCollector) Collect(ch chan<- Metric) {
c.collectFn(ch)
}
// Describe returns all descriptions of the collector.
func (c *processCollector) Describe(ch chan<- *Desc) {
c.describeFn(ch)
}
func (c *processCollector) reportError(ch chan<- Metric, desc *Desc, err error) {
if !c.reportErrors {
return
}
if desc == nil {
desc = NewInvalidDesc(err)
}
ch <- NewInvalidMetric(desc, err)
}
// NewPidFileFn returns a function that retrieves a pid from the specified file.
// It is meant to be used for the PidFn field in ProcessCollectorOpts.
func NewPidFileFn(pidFilePath string) func() (int, error) {
return func() (int, error) {
content, err := os.ReadFile(pidFilePath)
if err != nil {
return 0, fmt.Errorf("can't read pid file %q: %w", pidFilePath, err)
}
pid, err := strconv.Atoi(strings.TrimSpace(string(content)))
if err != nil {
return 0, fmt.Errorf("can't parse pid file %q: %w", pidFilePath, err)
}
return pid, nil
}
}

Some files were not shown because too many files have changed in this diff Show More