Compare commits

...
Author SHA1 Message Date
dev_agentandClaude Opus 5.5 70f545152a Validate decoded photo size and require a photos row to serve
CI / ci (push) Successful in 1m37s
CI / ci (pull_request) Successful in 1m37s
decodePhoto checked the GIF logical screen size, but gif.Decode returns
the first frame, which may be smaller; a 400x400 screen with a 1x1 first
frame passed and was scaled. The short-side check now measures the
decoded image's bounds. The pixel cap still comes from the header, before
decoding.

OpenPhoto served any file whose name was a canonical ULID and stored
size, with no photos row required. It now looks the id up with the new
GetPhoto query and reports an id with no row as fs.ErrNotExist, so
GET /media/photos/{id}-{size}.jpg is 404 for it even when a file exists.
OpenPhoto takes the request context for the lookup.

The media test fixture records a photos row for its served photo and
adds an orphan photo whose files exist without a row; both its sizes
must be 404.

Maestro task: 8a432108-d3ba-4013-ae8c-11963981d641

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 14:10:54 -04:00
dev_agentandClaude Opus 5.5 0347e11b7d Add photo cache and /media/photos endpoint
CI / ci (push) Successful in 1m38s
CI / ci (pull_request) Successful in 1m39s
recipes.PhotoCache.CachePhoto downloads an image through the safe
Fetcher's new FetchImage (JPEG/PNG/WebP/GIF content types only, 8 MB
cap, uncached), decodes it (GIF first frame), refuses images under
300 px on the short side, scales to 1200 px and 400 px on the long side
with golang.org/x/image/draw, encodes JPEG quality 82 under
$MEALQUEST_DATA_DIR/media/photos/{id}-{size}.jpg, deduplicates by the
SHA-256 of the original bytes and records a photos row.

GET /media/photos/{id}-{400|1200}.jpg serves a stored size behind the
session with Cache-Control "private, max-age=31536000, immutable";
anything unknown is 404.

Adds golang.org/x/image v0.45.0 (approved; v0.46.0 needs Go 1.26).

Maestro task: 8a432108-d3ba-4013-ae8c-11963981d641

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 13:59:02 -04:00
dev_agent 2fb7807af3 Search providers: SearXNG and Brave with ranking (#67)
CI / ci (push) Successful in 1m37s
Co-authored-by: dev_agent <dev_agent@cubecraftlabs.com>
2026-10-05 13:41:48 -04:00
24 changed files with 2095 additions and 26 deletions
+7 -1
View File
@@ -19,6 +19,7 @@ import (
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/frontend"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/household"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/jobs"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/recipes"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/secretbox"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/server"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
@@ -72,6 +73,11 @@ func run(cfg config.Config, logger *slog.Logger) error {
return err
}
photos, err := recipes.NewPhotoCache(recipes.NewFetcher(), db, cfg.DataDir)
if err != nil {
return err
}
sessions, err := auth.NewSessionManager(db, cfg.SessionIdle, cfg.SessionMax)
if err != nil {
return err
@@ -88,7 +94,7 @@ func run(cfg config.Config, logger *slog.Logger) error {
return err
}
router, err := server.NewRouter(logger, version, cfg.TrustedProxy, authFlow, sessions, settings, runner, frontend.Assets())
router, err := server.NewRouter(logger, version, cfg.TrustedProxy, authFlow, sessions, settings, runner, photos, frontend.Assets())
if err != nil {
return err
}
+36 -2
View File
@@ -140,7 +140,11 @@ automatic** (D-021).
It requires the session cookie and, for non-GET requests, the
`X-MealQuest-CSRF` header.
- `/auth/*` — the OIDC flow.
- `/media/*` — cached photos.
- `/media/*` — cached photos, behind the session cookie.
`GET /media/photos/{id}-{400|1200}.jpg` serves a stored size of a photo
with a `photos` row, with
`Cache-Control: private, max-age=31536000, immutable`; anything else is
404.
- `/health` — liveness.
- Everything else serves the embedded PWA with SPA fallback.
@@ -239,8 +243,38 @@ automatic** (D-021).
with two is `ErrManyRecipes`; a shape schema.org does not allow — such as
a non-ISO-8601 duration like "20 minutes" or an empty one, or an image
`@id` no node has — is `ErrMalformedRecipe`, never guessed at.
- Recipe search (`recipes.SearchProvider`: `Search(ctx, query, n)` →
`[]Candidate{URL, Title, Snippet}`, n 1–20) appends " recipe" to a query
without that word. SearXNG is called at
`{MEALQUEST_SEARXNG_URL}/search?q=…&format=json&categories=general` —
directly, not through the Fetcher, since this configured endpoint may be
on the internal network; Brave calls the Brave Web Search API with the
key decrypted from `recipe_search_settings.api_key_ct`
(`recipes.NewSearchProvider`). Redirects are not followed; 401/403 is
`ErrSearchRefused`, 429 `ErrSearchRateLimited`, other non-2xx a
`SearchStatusError`, and a body that is not the provider's JSON (SearXNG
without a `results` array; Brave without `"type": "search"` or with a
`web` object lacking `results` — Brave omits `web` when nothing matched),
over 1 MB or with a result lacking an absolute http(s) URL
`ErrSearchResponse`.
- Ranking (`recipes.Rank`) fetches and extracts the top 5 candidates
concurrently, rejects any whose recipe lacks ingredients or an image, and
orders the rest by the token similarity of the recipe's title to the dish
name (`harness.Similarity`, 0–1) plus a quarter for each of ingredients,
image, steps and total minutes; ties keep search order. No unranked
candidate is dropped silently: when some are ranked, they are returned
with `ErrCandidatesUnranked`; when none are, the error is
`ErrNoUsableRecipe`. Either is joined with each unranked candidate's URL
and reason (fetch failure, no valid recipe, missing ingredients or
image), so the caller decides per item what a failure means.
- Photos are cached locally (`photos`) and processed with
`golang.org/x/image`.
`golang.org/x/image`. `recipes.PhotoCache.CachePhoto` downloads through
the Fetcher's `FetchImage` (JPEG, PNG, WebP or GIF content types, 8 MB
cap, not cached), decodes the image (a GIF's first frame), refuses a
decoded image under 300 px on its short side, scales it so its long side is 1200 px and
400 px, and stores each as JPEG quality 82 at
`$MEALQUEST_DATA_DIR/media/photos/{id}-{size}.jpg`. Images are
deduplicated by the SHA-256 of the downloaded bytes.
## Scheduler
+1
View File
@@ -6,6 +6,7 @@ require (
github.com/coreos/go-oidc/v3 v3.21.0
github.com/go-chi/chi/v5 v5.3.1
github.com/oklog/ulid/v2 v2.1.2
golang.org/x/image v0.45.0
golang.org/x/oauth2 v0.36.0
modernc.org/sqlite v1.59.0
)
+2
View File
@@ -21,6 +21,8 @@ github.com/oklog/ulid/v2 v2.1.2/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNs
github.com/pborman/getopt v0.0.0-20170112200414-7148bc3a4c30/go.mod h1:85jBQOZwpVEaDAr341tbn15RS4fCAsIst0qp7i8ex1o=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
golang.org/x/image v0.45.0 h1:FMb1nTbH5H9vF55SriQHgFw5GnNL9Jg6L25BwXKzhB0=
golang.org/x/image v0.45.0/go.mod h1:n62x/7RqlwXDvGsSU4u6IUTUf6KghUZ9Bt7cG/T9Fx4=
golang.org/x/mod v0.38.0 h1:MECBjubtXD7yj4HrhIUcywNaGeNVUdfVnxmPajOk4yk=
golang.org/x/mod v0.38.0/go.mod h1:V6Xz0pq8TQ3dGqVQ1FVHuelZpAL0uNhSkk9ogYP3c40=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
+8 -1
View File
@@ -53,7 +53,14 @@ func Normalize(name string) string {
// similarity of their normalized forms is at least SimilarThreshold. A name
// with no words left after normalizing is similar to nothing.
func Similar(a, b string) bool {
return similarity(tokenSet(a), tokenSet(b)) >= SimilarThreshold
return Similarity(a, b) >= SimilarThreshold
}
// Similarity is the token-set (Jaccard) similarity of a's and b's normalized
// forms, from 0 (no shared word) to 1 (the same words). A name with no words
// left after normalizing scores 0 against everything.
func Similarity(a, b string) float64 {
return similarity(tokenSet(a), tokenSet(b))
}
// tokenSet returns the distinct words of name's normalized form.
+16
View File
@@ -114,6 +114,22 @@ func TestSimilar(t *testing.T) {
}
}
func TestSimilarity(t *testing.T) {
for _, tc := range []struct {
a, b string
want float64
}{
{"Easy Chicken Curry", "chicken curry", 1},
{"Thai Green Curry", "Chicken curry", 0.25},
{"Beef Stew", "Chicken curry", 0},
{"The Recipe", "chicken curry", 0},
} {
if got := Similarity(tc.a, tc.b); got != tc.want {
t.Errorf("Similarity(%q, %q) = %v, want %v", tc.a, tc.b, got, tc.want)
}
}
}
func TestDeduper(t *testing.T) {
d := NewDeduper(DedupeSources{
Regulars: []string{"Beef Tacos", "Spaghetti Bolognese"},
+22
View File
@@ -59,6 +59,28 @@ func (q *Queries) CreatePhoto(ctx context.Context, arg CreatePhotoParams) (Photo
return i, err
}
const getPhoto = `-- name: GetPhoto :one
SELECT id, source_url, local_path, width, height, sha256, fetched_at, created_at, updated_at FROM photos
WHERE id = ?
`
func (q *Queries) GetPhoto(ctx context.Context, id string) (Photo, error) {
row := q.db.QueryRowContext(ctx, getPhoto, id)
var i Photo
err := row.Scan(
&i.ID,
&i.SourceUrl,
&i.LocalPath,
&i.Width,
&i.Height,
&i.Sha256,
&i.FetchedAt,
&i.CreatedAt,
&i.UpdatedAt,
)
return i, err
}
const getPhotoBySHA256 = `-- name: GetPhotoBySHA256 :one
SELECT id, source_url, local_path, width, height, sha256, fetched_at, created_at, updated_at FROM photos
WHERE sha256 = ?
+1
View File
@@ -43,6 +43,7 @@ type Querier interface {
// Returns sql.ErrNoRows until the row exists.
GetIntegrationsGoogle(ctx context.Context) (IntegrationsGoogle, error)
GetJob(ctx context.Context, id string) (Job, error)
GetPhoto(ctx context.Context, id string) (Photo, error)
// Finds stored image bytes by content hash. The schema does not make sha256
// unique, so the earliest-created (lowest id) match is returned.
GetPhotoBySHA256(ctx context.Context, sha256 string) (Photo, error)
+66 -14
View File
@@ -8,10 +8,12 @@ import (
"errors"
"fmt"
"io"
"mime"
"net"
"net/http"
"net/netip"
"net/url"
"slices"
"strings"
"sync"
"time"
@@ -23,6 +25,8 @@ const (
// MaxBodyBytes caps a fetched body. A larger page is an error, never
// silently truncated: a cut-off recipe page would parse as a wrong one.
MaxBodyBytes = 2 << 20
// MaxImageBytes caps a fetched image body, as MaxBodyBytes caps a page.
MaxImageBytes = 8 << 20
// MaxRedirects is how many redirects a fetch follows; each hop is
// re-checked for scheme and address.
MaxRedirects = 3
@@ -51,6 +55,25 @@ var (
ErrTooLarge = errors.New("recipes: body exceeds size cap")
// ErrTimeout is returned when a fetch runs past Timeout.
ErrTimeout = errors.New("recipes: fetch timed out")
// ErrNotImage is returned by FetchImage for a response whose
// Content-Type is not one of ImageContentTypes.
ErrNotImage = errors.New("recipes: response is not a supported image")
)
// ImageContentTypes are the media types FetchImage accepts.
var ImageContentTypes = []string{"image/jpeg", "image/png", "image/webp", "image/gif"}
// fetchKind is what a request asks for: its Accept header, the body cap and
// the media types allowed, where nil allows any.
type fetchKind struct {
accept string
maxBytes int64
contentTypes []string
}
var (
pageKind = fetchKind{accept: "text/html,application/xhtml+xml", maxBytes: MaxBodyBytes}
imageKind = fetchKind{accept: strings.Join(ImageContentTypes, ","), maxBytes: MaxImageBytes, contentTypes: ImageContentTypes}
)
// StatusError is returned for a final response whose status is not 2xx.
@@ -163,27 +186,49 @@ func (f *Fetcher) Fetch(ctx context.Context, rawURL string) (Page, error) {
if page, ok := f.cached(key); ok {
return page, nil
}
fetchCtx, cancel := context.WithTimeout(ctx, f.timeout)
defer cancel()
page, err := f.get(fetchCtx, key)
page, err := f.fetch(ctx, key, pageKind)
if err != nil {
if ctx.Err() == nil && errors.Is(fetchCtx.Err(), context.DeadlineExceeded) {
return Page{}, fmt.Errorf("%w after %s: %s", ErrTimeout, f.timeout, key)
}
return Page{}, err
}
f.store(key, page)
return page, nil
}
func (f *Fetcher) get(ctx context.Context, rawURL string) (Page, error) {
// FetchImage returns the image at rawURL under the same safeguards as
// Fetch, but with a body cap of MaxImageBytes, and only for a response
// whose Content-Type is one of ImageContentTypes. Images are not cached.
func (f *Fetcher) FetchImage(ctx context.Context, rawURL string) (Page, error) {
u, err := url.Parse(rawURL)
if err != nil {
return Page{}, fmt.Errorf("recipes: parse URL: %w", err)
}
if err := checkURL(u); err != nil {
return Page{}, err
}
return f.fetch(ctx, u.String(), imageKind)
}
// fetch gets rawURL as kind within the Fetcher's timeout.
func (f *Fetcher) fetch(ctx context.Context, rawURL string, kind fetchKind) (Page, error) {
fetchCtx, cancel := context.WithTimeout(ctx, f.timeout)
defer cancel()
page, err := f.get(fetchCtx, rawURL, kind)
if err != nil {
if ctx.Err() == nil && errors.Is(fetchCtx.Err(), context.DeadlineExceeded) {
return Page{}, fmt.Errorf("%w after %s: %s", ErrTimeout, f.timeout, rawURL)
}
return Page{}, err
}
return page, nil
}
func (f *Fetcher) get(ctx context.Context, rawURL string, kind fetchKind) (Page, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)
if err != nil {
return Page{}, fmt.Errorf("recipes: build request: %w", err)
}
req.Header.Set("User-Agent", UserAgent)
req.Header.Set("Accept", "text/html,application/xhtml+xml")
req.Header.Set("Accept", kind.accept)
resp, err := f.client.Do(req)
if err != nil {
return Page{}, err
@@ -193,17 +238,24 @@ func (f *Fetcher) get(ctx context.Context, rawURL string) (Page, error) {
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return Page{}, &StatusError{URL: final, Status: resp.StatusCode}
}
if resp.ContentLength > MaxBodyBytes {
contentType := resp.Header.Get("Content-Type")
if kind.contentTypes != nil {
mediaType, _, err := mime.ParseMediaType(contentType)
if err != nil || !slices.Contains(kind.contentTypes, mediaType) {
return Page{}, fmt.Errorf("%w: %s has Content-Type %q", ErrNotImage, final, contentType)
}
}
if resp.ContentLength > kind.maxBytes {
return Page{}, fmt.Errorf("%w: %s declares %d bytes", ErrTooLarge, final, resp.ContentLength)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, MaxBodyBytes+1))
body, err := io.ReadAll(io.LimitReader(resp.Body, kind.maxBytes+1))
if err != nil {
return Page{}, fmt.Errorf("recipes: read %s: %w", final, err)
}
if len(body) > MaxBodyBytes {
return Page{}, fmt.Errorf("%w: %s is over %d bytes", ErrTooLarge, final, MaxBodyBytes)
if int64(len(body)) > kind.maxBytes {
return Page{}, fmt.Errorf("%w: %s is over %d bytes", ErrTooLarge, final, kind.maxBytes)
}
return Page{URL: final, ContentType: resp.Header.Get("Content-Type"), Body: body}, nil
return Page{URL: final, ContentType: contentType, Body: body}, nil
}
// cached returns the page cached under key if it is younger than CacheTTL.
+254
View File
@@ -0,0 +1,254 @@
package recipes
import (
"bytes"
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"errors"
"fmt"
"image"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"math"
"os"
"path"
"path/filepath"
"slices"
"strconv"
"sync"
"github.com/oklog/ulid/v2"
"golang.org/x/image/draw"
"golang.org/x/image/webp"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
)
const (
// PhotoDir is where cached photos are stored, relative to the data
// directory.
PhotoDir = "media/photos"
// PhotoMinShortSide is the least a photo's shorter side may measure, in
// pixels.
PhotoMinShortSide = 300
// PhotoMaxPixels bounds the pixels a photo may decode to, so a small
// compressed file cannot claim gigabytes of memory.
PhotoMaxPixels = 40_000_000
// PhotoQuality is the JPEG quality of every stored size.
PhotoQuality = 82
)
// PhotoSizes are the long-side lengths, in pixels, each photo is stored at.
var PhotoSizes = []int{1200, 400}
var (
// ErrPhotoTooSmall is returned for an image whose shorter side is under
// PhotoMinShortSide.
ErrPhotoTooSmall = errors.New("recipes: photo is too small")
// ErrPhotoTooLarge is returned for an image over PhotoMaxPixels.
ErrPhotoTooLarge = errors.New("recipes: photo has too many pixels")
// ErrPhotoFormat is returned for bytes that are not a JPEG, PNG, WebP
// or GIF image.
ErrPhotoFormat = errors.New("recipes: photo is not a JPEG, PNG, WebP or GIF image")
)
// photoDecoders are the formats a photo may be in. A GIF decodes to its
// first frame.
var photoDecoders = map[string]func([]byte) (image.Image, error){
"jpeg": func(b []byte) (image.Image, error) { return jpeg.Decode(bytes.NewReader(b)) },
"png": func(b []byte) (image.Image, error) { return png.Decode(bytes.NewReader(b)) },
"gif": func(b []byte) (image.Image, error) { return gif.Decode(bytes.NewReader(b)) },
"webp": func(b []byte) (image.Image, error) { return webp.Decode(bytes.NewReader(b)) },
}
// PhotoCache downloads photos through a Fetcher and stores each at every
// PhotoSizes length as {id}-{size}.jpg under PhotoDir, with a photos row.
// Photos are deduplicated by the SHA-256 of the downloaded bytes. A
// PhotoCache is safe for concurrent use; photos are processed one at a time.
type PhotoCache struct {
fetcher *Fetcher
store *store.Store
// dir is the absolute photo directory.
dir string
// mu serialises the dedupe lookup, processing and insert, so one image
// is never stored twice.
mu sync.Mutex
}
// NewPhotoCache returns a PhotoCache storing photos under dataDir, creating
// the photo directory if it is absent.
func NewPhotoCache(fetcher *Fetcher, st *store.Store, dataDir string) (*PhotoCache, error) {
if fetcher == nil {
return nil, errors.New("recipes: nil fetcher")
}
if st == nil {
return nil, errors.New("recipes: nil store")
}
dir, err := filepath.Abs(filepath.Join(dataDir, filepath.FromSlash(PhotoDir)))
if err != nil {
return nil, fmt.Errorf("recipes: resolve photo directory: %w", err)
}
if err := os.MkdirAll(dir, 0o750); err != nil {
return nil, fmt.Errorf("recipes: create photo directory: %w", err)
}
return &PhotoCache{fetcher: fetcher, store: st, dir: dir}, nil
}
// CachePhoto downloads the image at rawURL and returns its photos row. An
// image whose bytes are already cached returns the existing row. Otherwise
// it is decoded, refused if its shorter side is under PhotoMinShortSide,
// scaled so its long side is each of PhotoSizes, encoded as JPEG and
// recorded. The row's width and height are the original image's, and its
// local_path is the largest size's file relative to the data directory.
func (c *PhotoCache) CachePhoto(ctx context.Context, rawURL string) (db.Photo, error) {
page, err := c.fetcher.FetchImage(ctx, rawURL)
if err != nil {
return db.Photo{}, err
}
digest := sha256.Sum256(page.Body)
sum := hex.EncodeToString(digest[:])
c.mu.Lock()
defer c.mu.Unlock()
existing, err := db.New(c.store.Reader).GetPhotoBySHA256(ctx, sum)
if err == nil {
return existing, nil
}
if !errors.Is(err, sql.ErrNoRows) {
return db.Photo{}, fmt.Errorf("recipes: look up photo: %w", err)
}
img, err := decodePhoto(page.Body)
if err != nil {
return db.Photo{}, fmt.Errorf("%w: %s", err, page.URL)
}
id := store.NewID()
var written []string
removeWritten := func() error {
var errs []error
for _, p := range written {
errs = append(errs, os.Remove(p))
}
return errors.Join(errs...)
}
for _, size := range PhotoSizes {
p := c.path(id, size)
if err := writeJPEG(p, scaleLongSide(img, size)); err != nil {
return db.Photo{}, errors.Join(err, removeWritten())
}
written = append(written, p)
}
now := store.Now().String()
bounds := img.Bounds()
photo, err := db.New(c.store.Writer).CreatePhoto(ctx, db.CreatePhotoParams{
ID: id,
SourceUrl: page.URL,
LocalPath: path.Join(PhotoDir, photoFileName(id, PhotoSizes[0])),
Width: int64(bounds.Dx()),
Height: int64(bounds.Dy()),
Sha256: sum,
FetchedAt: now,
CreatedAt: now,
UpdatedAt: now,
})
if err != nil {
return db.Photo{}, errors.Join(fmt.Errorf("recipes: record photo: %w", err), removeWritten())
}
return photo, nil
}
// OpenPhoto opens the stored file of photo id at size. An id that is not a
// canonical ULID, a size not in PhotoSizes, an id with no photos row, or a
// photo whose file is not stored is an error matching fs.ErrNotExist.
func (c *PhotoCache) OpenPhoto(ctx context.Context, id string, size int) (*os.File, error) {
parsed, err := ulid.ParseStrict(id)
if err != nil || parsed.String() != id || !slices.Contains(PhotoSizes, size) {
return nil, fmt.Errorf("recipes: photo %q at size %d: %w", id, size, fs.ErrNotExist)
}
if _, err := db.New(c.store.Reader).GetPhoto(ctx, id); errors.Is(err, sql.ErrNoRows) {
return nil, fmt.Errorf("recipes: photo %q: %w", id, fs.ErrNotExist)
} else if err != nil {
return nil, fmt.Errorf("recipes: look up photo %q: %w", id, err)
}
return os.Open(c.path(id, size))
}
func (c *PhotoCache) path(id string, size int) string {
return filepath.Join(c.dir, photoFileName(id, size))
}
func photoFileName(id string, size int) string {
return id + "-" + strconv.Itoa(size) + ".jpg"
}
// decodePhoto decodes a JPEG, PNG, WebP or GIF image, checking its pixel
// count from the header before decoding the pixels, and the shorter side of
// the decoded image: a GIF's first frame may be smaller than its screen.
func decodePhoto(b []byte) (image.Image, error) {
cfg, format, err := image.DecodeConfig(bytes.NewReader(b))
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrPhotoFormat, err)
}
decode, ok := photoDecoders[format]
if !ok {
return nil, fmt.Errorf("%w: format %q", ErrPhotoFormat, format)
}
if int64(cfg.Width)*int64(cfg.Height) > PhotoMaxPixels {
return nil, fmt.Errorf("%w: %dx%d", ErrPhotoTooLarge, cfg.Width, cfg.Height)
}
img, err := decode(b)
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrPhotoFormat, err)
}
if size := img.Bounds().Size(); min(size.X, size.Y) < PhotoMinShortSide {
return nil, fmt.Errorf("%w: %dx%d, shorter side under %d px", ErrPhotoTooSmall, size.X, size.Y, PhotoMinShortSide)
}
return img, nil
}
// scaleLongSide returns img scaled so its longer side is long pixels,
// keeping its aspect ratio. Transparent areas become white, since JPEG has
// no alpha.
func scaleLongSide(img image.Image, long int) *image.RGBA {
b := img.Bounds()
w, h := b.Dx(), b.Dy()
dw, dh := long, long
if w >= h {
dh = max(1, int(math.Round(float64(h)*float64(long)/float64(w))))
} else {
dw = max(1, int(math.Round(float64(w)*float64(long)/float64(h))))
}
dst := image.NewRGBA(image.Rect(0, 0, dw, dh))
draw.Draw(dst, dst.Bounds(), image.White, image.Point{}, draw.Src)
draw.CatmullRom.Scale(dst, dst.Bounds(), img, b, draw.Over, nil)
return dst
}
// writeJPEG encodes img at PhotoQuality into a temporary file beside path
// and renames it into place, so a reader never sees a partial file.
func writeJPEG(path string, img image.Image) error {
f, err := os.CreateTemp(filepath.Dir(path), ".photo-*.tmp")
if err != nil {
return fmt.Errorf("recipes: create photo file: %w", err)
}
tmp := f.Name()
err = jpeg.Encode(f, img, &jpeg.Options{Quality: PhotoQuality})
if err == nil {
err = f.Sync()
}
err = errors.Join(err, f.Close())
if err == nil {
err = os.Rename(tmp, path)
}
if err != nil {
return errors.Join(fmt.Errorf("recipes: write photo %s: %w", filepath.Base(path), err), os.Remove(tmp))
}
return nil
}
+438
View File
@@ -0,0 +1,438 @@
package recipes
import (
"bytes"
"context"
"errors"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/migrations"
)
// photoServer serves each path's body with its Content-Type and counts
// requests.
type photoServer struct {
files map[string]photoFile
requests atomic.Int32
}
type photoFile struct {
contentType string
body []byte
}
func (s *photoServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.requests.Add(1)
f, ok := s.files[r.URL.Path]
if !ok {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", f.contentType)
w.Write(f.body)
}
// newTestPhotoCache returns a PhotoCache over files, a migrated temp
// database and a temp data directory.
func newTestPhotoCache(t *testing.T, files map[string]photoFile) (*PhotoCache, *store.Store, string, *photoServer) {
t.Helper()
srv := &photoServer{files: files}
env := newTestEnv(t, srv, publicAnswers())
installClock(env.f)
dataDir := t.TempDir()
st, err := store.Open(t.Context(), dataDir)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
if err := st.Close(); err != nil {
t.Error(err)
}
})
if err := store.Migrate(t.Context(), st.Writer, migrations.FS); err != nil {
t.Fatal(err)
}
cache, err := NewPhotoCache(env.f, st, dataDir)
if err != nil {
t.Fatal(err)
}
return cache, st, dataDir, srv
}
// gradient is a w×h opaque image.
func gradient(w, h int) *image.RGBA {
img := image.NewRGBA(image.Rect(0, 0, w, h))
for y := range h {
for x := range w {
img.Set(x, y, color.RGBA{uint8(x * 255 / w), uint8(y * 255 / h), 128, 255})
}
}
return img
}
func encodeJPEG(t *testing.T, img image.Image) []byte {
t.Helper()
var b bytes.Buffer
if err := jpeg.Encode(&b, img, nil); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func encodePNG(t *testing.T, img image.Image) []byte {
t.Helper()
var b bytes.Buffer
if err := png.Encode(&b, img); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
// twoFrameGIF is a w×h animated GIF whose first frame is red and second
// blue.
func twoFrameGIF(t *testing.T, w, h int) []byte {
t.Helper()
frame := func(c color.Color) *image.Paletted {
p := image.NewPaletted(image.Rect(0, 0, w, h), color.Palette{c})
return p
}
var b bytes.Buffer
anim := &gif.GIF{
Image: []*image.Paletted{frame(color.RGBA{255, 0, 0, 255}), frame(color.RGBA{0, 0, 255, 255})},
Delay: []int{10, 10},
}
if err := gif.EncodeAll(&b, anim); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func readTestdata(t *testing.T, name string) []byte {
t.Helper()
b, err := os.ReadFile(filepath.Join("testdata", name))
if err != nil {
t.Fatal(err)
}
return b
}
// storedSize decodes the stored JPEG of id at size and returns it.
func storedSize(t *testing.T, dataDir, id string, size int) image.Image {
t.Helper()
b, err := os.ReadFile(filepath.Join(dataDir, "media", "photos", photoFileName(id, size)))
if err != nil {
t.Fatalf("read stored photo: %v", err)
}
img, format, err := image.Decode(bytes.NewReader(b))
if err != nil || format != "jpeg" {
t.Fatalf("stored photo is %q, %v; want a JPEG", format, err)
}
return img
}
func TestCachePhotoFormatsAndSizes(t *testing.T) {
t.Parallel()
files := map[string]photoFile{
"/landscape.jpg": {"image/jpeg", encodeJPEG(t, gradient(800, 600))},
"/portrait.png": {"image/png", encodePNG(t, gradient(600, 900))},
"/photo.webp": {"image/webp", readTestdata(t, "photo.webp")},
"/anim.gif": {"image/gif; charset=binary", twoFrameGIF(t, 300, 450)},
"/square.png": {"image/png", encodePNG(t, gradient(1500, 1500))},
}
cache, _, dataDir, _ := newTestPhotoCache(t, files)
tests := []struct {
path string
width, height int64
// sizes maps each stored size to its expected dimensions.
sizes map[int]image.Point
}{
{"/landscape.jpg", 800, 600, map[int]image.Point{1200: {1200, 900}, 400: {400, 300}}},
{"/portrait.png", 600, 900, map[int]image.Point{1200: {800, 1200}, 400: {267, 400}}},
{"/photo.webp", 640, 480, map[int]image.Point{1200: {1200, 900}, 400: {400, 300}}},
{"/anim.gif", 300, 450, map[int]image.Point{1200: {800, 1200}, 400: {267, 400}}},
{"/square.png", 1500, 1500, map[int]image.Point{1200: {1200, 1200}, 400: {400, 400}}},
}
for _, tc := range tests {
photo, err := cache.CachePhoto(t.Context(), "https://example.com"+tc.path)
if err != nil {
t.Fatalf("CachePhoto %s: %v", tc.path, err)
}
if photo.Width != tc.width || photo.Height != tc.height {
t.Errorf("%s: row is %dx%d, want %dx%d", tc.path, photo.Width, photo.Height, tc.width, tc.height)
}
if photo.SourceUrl != "https://example.com"+tc.path || len(photo.Sha256) != 64 {
t.Errorf("%s: row = %+v", tc.path, photo)
}
if want := "media/photos/" + photo.ID + "-1200.jpg"; photo.LocalPath != want {
t.Errorf("%s: local_path = %q, want %q", tc.path, photo.LocalPath, want)
}
for size, want := range tc.sizes {
img := storedSize(t, dataDir, photo.ID, size)
if got := img.Bounds().Size(); got != want {
t.Errorf("%s at %d: stored %v, want %v", tc.path, size, got, want)
}
}
}
}
func TestCachePhotoGIFUsesFirstFrame(t *testing.T) {
t.Parallel()
cache, _, dataDir, _ := newTestPhotoCache(t, map[string]photoFile{
"/anim.gif": {"image/gif", twoFrameGIF(t, 400, 400)},
})
photo, err := cache.CachePhoto(t.Context(), "https://example.com/anim.gif")
if err != nil {
t.Fatalf("CachePhoto: %v", err)
}
r, g, b, _ := storedSize(t, dataDir, photo.ID, 400).At(200, 200).RGBA()
if r>>8 < 200 || g>>8 > 50 || b>>8 > 50 {
t.Errorf("centre pixel = (%d,%d,%d), want the first frame's red", r>>8, g>>8, b>>8)
}
}
func TestCachePhotoRejectsSmallGIFFirstFrame(t *testing.T) {
t.Parallel()
// A 400x400 screen whose first frame is 1x1: the frame is what would be
// stored, so it is what must measure PhotoMinShortSide.
var b bytes.Buffer
anim := &gif.GIF{
Image: []*image.Paletted{image.NewPaletted(image.Rect(0, 0, 1, 1), color.Palette{color.Black})},
Delay: []int{10},
Config: image.Config{ColorModel: color.Palette{color.Black}, Width: 400, Height: 400},
}
if err := gif.EncodeAll(&b, anim); err != nil {
t.Fatal(err)
}
cache, st, dataDir, _ := newTestPhotoCache(t, map[string]photoFile{
"/speck.gif": {"image/gif", b.Bytes()},
})
if _, err := cache.CachePhoto(t.Context(), "https://example.com/speck.gif"); !errors.Is(err, ErrPhotoTooSmall) {
t.Errorf("CachePhoto error = %v, want ErrPhotoTooSmall", err)
}
if n := countPhotos(t, st); n != 0 {
t.Errorf("%d photos recorded for a refused image", n)
}
if names := photoFiles(t, dataDir); len(names) != 0 {
t.Errorf("files stored for a refused image: %v", names)
}
}
func TestCachePhotoFillsTransparencyWhite(t *testing.T) {
t.Parallel()
cache, _, dataDir, _ := newTestPhotoCache(t, map[string]photoFile{
"/clear.png": {"image/png", encodePNG(t, image.NewNRGBA(image.Rect(0, 0, 400, 400)))},
})
photo, err := cache.CachePhoto(t.Context(), "https://example.com/clear.png")
if err != nil {
t.Fatalf("CachePhoto: %v", err)
}
r, g, b, _ := storedSize(t, dataDir, photo.ID, 400).At(200, 200).RGBA()
if r>>8 < 250 || g>>8 < 250 || b>>8 < 250 {
t.Errorf("transparent pixel stored as (%d,%d,%d), want white", r>>8, g>>8, b>>8)
}
}
func countPhotos(t *testing.T, st *store.Store) int {
t.Helper()
var n int
if err := st.Reader.QueryRowContext(context.Background(), "SELECT count(*) FROM photos").Scan(&n); err != nil {
t.Fatal(err)
}
return n
}
func photoFiles(t *testing.T, dataDir string) []string {
t.Helper()
entries, err := os.ReadDir(filepath.Join(dataDir, "media", "photos"))
if err != nil {
t.Fatal(err)
}
var names []string
for _, e := range entries {
names = append(names, e.Name())
}
return names
}
func TestCachePhotoRejectsTooSmall(t *testing.T) {
t.Parallel()
cache, st, dataDir, _ := newTestPhotoCache(t, map[string]photoFile{
"/narrow.png": {"image/png", encodePNG(t, gradient(299, 1000))},
"/short.jpg": {"image/jpeg", encodeJPEG(t, gradient(1000, 299))},
"/edge.png": {"image/png", encodePNG(t, gradient(300, 1000))},
})
for _, p := range []string{"/narrow.png", "/short.jpg"} {
if _, err := cache.CachePhoto(t.Context(), "https://example.com"+p); !errors.Is(err, ErrPhotoTooSmall) {
t.Errorf("CachePhoto %s error = %v, want ErrPhotoTooSmall", p, err)
}
}
if n := countPhotos(t, st); n != 0 {
t.Errorf("%d photos recorded for refused images", n)
}
if names := photoFiles(t, dataDir); len(names) != 0 {
t.Errorf("files stored for refused images: %v", names)
}
if _, err := cache.CachePhoto(t.Context(), "https://example.com/edge.png"); err != nil {
t.Errorf("CachePhoto with a 300 px short side: %v", err)
}
}
func TestCachePhotoRejectsNonImages(t *testing.T) {
t.Parallel()
cache, st, _, _ := newTestPhotoCache(t, map[string]photoFile{
"/page.html": {"text/html", encodePNG(t, gradient(400, 400))},
"/fake.jpg": {"image/jpeg", []byte("<html>not a picture</html>")},
})
if _, err := cache.CachePhoto(t.Context(), "https://example.com/page.html"); !errors.Is(err, ErrNotImage) {
t.Errorf("text/html error = %v, want ErrNotImage", err)
}
if _, err := cache.CachePhoto(t.Context(), "https://example.com/fake.jpg"); !errors.Is(err, ErrPhotoFormat) {
t.Errorf("undecodable image error = %v, want ErrPhotoFormat", err)
}
if n := countPhotos(t, st); n != 0 {
t.Errorf("%d photos recorded", n)
}
}
func TestCachePhotoDeduplicatesBySHA256(t *testing.T) {
t.Parallel()
body := encodeJPEG(t, gradient(640, 480))
cache, st, dataDir, srv := newTestPhotoCache(t, map[string]photoFile{
"/a.jpg": {"image/jpeg", body},
"/b.jpg": {"image/jpeg", body},
"/other.jpg": {"image/jpeg", encodeJPEG(t, gradient(480, 640))},
})
first, err := cache.CachePhoto(t.Context(), "https://example.com/a.jpg")
if err != nil {
t.Fatalf("CachePhoto a: %v", err)
}
again, err := cache.CachePhoto(t.Context(), "https://example.com/b.jpg")
if err != nil {
t.Fatalf("CachePhoto b: %v", err)
}
if again != first {
t.Errorf("same bytes at another URL = %+v, want the first row %+v", again, first)
}
if n := srv.requests.Load(); n != 2 {
t.Errorf("%d downloads, want 2: each URL is fetched to hash it", n)
}
other, err := cache.CachePhoto(t.Context(), "https://example.com/other.jpg")
if err != nil {
t.Fatalf("CachePhoto other: %v", err)
}
if other.ID == first.ID || other.Sha256 == first.Sha256 {
t.Errorf("different bytes share a photo: %+v and %+v", first, other)
}
if n := countPhotos(t, st); n != 2 {
t.Errorf("%d photos recorded, want 2", n)
}
if names := photoFiles(t, dataDir); len(names) != 4 {
t.Errorf("stored files %v, want two sizes of two photos", names)
}
}
func TestOpenPhoto(t *testing.T) {
t.Parallel()
cache, _, dataDir, _ := newTestPhotoCache(t, map[string]photoFile{
"/a.png": {"image/png", encodePNG(t, gradient(400, 300))},
})
photo, err := cache.CachePhoto(t.Context(), "https://example.com/a.png")
if err != nil {
t.Fatalf("CachePhoto: %v", err)
}
for _, size := range PhotoSizes {
f, err := cache.OpenPhoto(t.Context(), photo.ID, size)
if err != nil {
t.Fatalf("OpenPhoto %d: %v", size, err)
}
f.Close()
}
// A file with no photos row is not a stored photo.
orphan := store.NewID()
if err := os.WriteFile(filepath.Join(dataDir, "media", "photos", orphan+"-400.jpg"), []byte("jpeg"), 0o600); err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
id string
size int
}{
{photo.ID, 800},
{store.NewID(), 400},
{orphan, 400},
{strings.ToLower(photo.ID), 400},
{"../" + photo.ID, 400},
{"", 400},
} {
if _, err := cache.OpenPhoto(t.Context(), tc.id, tc.size); !errors.Is(err, fs.ErrNotExist) {
t.Errorf("OpenPhoto(%q, %d) error = %v, want fs.ErrNotExist", tc.id, tc.size, err)
}
}
}
func TestFetchImageSizeCap(t *testing.T) {
t.Parallel()
env := newTestEnv(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
size := MaxImageBytes
if r.URL.Path != "/exact" {
size++
}
w.Header().Set("Content-Type", "image/jpeg")
w.Write(bytes.Repeat([]byte("a"), size))
}), publicAnswers())
installClock(env.f)
page, err := env.f.FetchImage(t.Context(), "https://example.com/exact")
if err != nil {
t.Fatalf("FetchImage at the cap: %v", err)
}
if len(page.Body) != MaxImageBytes {
t.Errorf("body is %d bytes, want %d", len(page.Body), MaxImageBytes)
}
if _, err := env.f.FetchImage(t.Context(), "https://example.com/over"); !errors.Is(err, ErrTooLarge) {
t.Errorf("FetchImage over the cap error = %v, want ErrTooLarge", err)
}
}
func TestFetchImageContentTypes(t *testing.T) {
t.Parallel()
var accept atomic.Value
env := newTestEnv(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
accept.Store(r.Header.Get("Accept"))
w.Header().Set("Content-Type", r.URL.Query().Get("type"))
w.Write([]byte("x"))
}), publicAnswers())
installClock(env.f)
typeURL := func(ct string) string { return "https://example.com/?type=" + url.QueryEscape(ct) }
for _, ct := range []string{"image/jpeg", "image/png", "image/webp", "image/gif", "IMAGE/PNG; q=1"} {
if _, err := env.f.FetchImage(t.Context(), typeURL(ct)); err != nil {
t.Errorf("FetchImage %q: %v", ct, err)
}
}
if got := accept.Load(); got != "image/jpeg,image/png,image/webp,image/gif" {
t.Errorf("Accept = %q", got)
}
for _, ct := range []string{"text/html", "image/svg+xml", "image/avif", "application/octet-stream", ""} {
if _, err := env.f.FetchImage(t.Context(), typeURL(ct)); !errors.Is(err, ErrNotImage) {
t.Errorf("FetchImage %q error = %v, want ErrNotImage", ct, err)
}
}
// Pages still fetch whatever the Content-Type.
if _, err := env.f.Fetch(t.Context(), typeURL("text/plain")); err != nil {
t.Errorf("Fetch: %v", err)
}
}
+130
View File
@@ -0,0 +1,130 @@
package recipes
import (
"context"
"errors"
"fmt"
"sort"
"sync"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/ai/harness"
)
// RankDepth is how many of the top search candidates Rank fetches and
// extracts.
const RankDepth = 5
var (
// ErrNoIngredients rejects a candidate whose recipe lists no
// ingredients.
ErrNoIngredients = errors.New("recipes: recipe has no ingredients")
// ErrNoImage rejects a candidate whose recipe has no image.
ErrNoImage = errors.New("recipes: recipe has no image")
// ErrNoUsableRecipe is returned when Rank ranks no candidate; it is
// joined with each candidate's reason.
ErrNoUsableRecipe = errors.New("recipes: no candidate holds a usable recipe")
// ErrCandidatesUnranked is returned, alongside the ranked candidates,
// when Rank ranks some candidates but not all; it is joined with each
// unranked candidate's reason.
ErrCandidatesUnranked = errors.New("recipes: some candidates were not ranked")
)
// PageFetcher fetches a candidate's page; *Fetcher satisfies it.
type PageFetcher interface {
Fetch(ctx context.Context, rawURL string) (Page, error)
}
// Ranked is a candidate whose page held a usable recipe, with its score.
type Ranked struct {
Candidate Candidate
Recipe Recipe
// Score is the token similarity of the recipe's title to the dish name
// (0 to 1, harness.Similarity) plus a quarter for each of ingredients,
// image, steps and total minutes present (0 to 1).
Score float64
}
// Rank fetches and extracts the first RankDepth candidates concurrently and
// returns those whose recipe has ingredients and an image, highest Score
// first; equal scores keep search order.
//
// Every candidate that is not ranked — its fetch failed, its page holds no
// valid recipe, or the recipe lacks ingredients or an image — is reported,
// never dropped silently: when some are ranked the error is
// ErrCandidatesUnranked, returned together with the ranked ones; when none
// are it is ErrNoUsableRecipe with no results. Either is joined with each
// unranked candidate's URL and reason. A cancelled ctx returns only its
// error.
func Rank(ctx context.Context, f PageFetcher, dish string, cands []Candidate) ([]Ranked, error) {
if len(cands) > RankDepth {
cands = cands[:RankDepth]
}
type outcome struct {
ranked Ranked
err error
}
outcomes := make([]outcome, len(cands))
var wg sync.WaitGroup
for i, c := range cands {
wg.Add(1)
go func() {
defer wg.Done()
r, err := rankOne(ctx, f, dish, c)
outcomes[i] = outcome{r, err}
}()
}
wg.Wait()
if err := ctx.Err(); err != nil {
return nil, err
}
var ranked []Ranked
var reasons []error
for i, o := range outcomes {
if o.err != nil {
reasons = append(reasons, fmt.Errorf("%s: %w", cands[i].URL, o.err))
continue
}
ranked = append(ranked, o.ranked)
}
if len(ranked) == 0 {
return nil, errors.Join(append([]error{ErrNoUsableRecipe}, reasons...)...)
}
sort.SliceStable(ranked, func(a, b int) bool { return ranked[a].Score > ranked[b].Score })
if len(reasons) > 0 {
return ranked, errors.Join(append([]error{ErrCandidatesUnranked}, reasons...)...)
}
return ranked, nil
}
// rankOne fetches, extracts and scores one candidate, or says why it is
// rejected.
func rankOne(ctx context.Context, f PageFetcher, dish string, c Candidate) (Ranked, error) {
page, err := f.Fetch(ctx, c.URL)
if err != nil {
return Ranked{}, err
}
r, err := Extract(page.Body, page.URL)
if err != nil {
return Ranked{}, err
}
if len(r.Ingredients) == 0 {
return Ranked{}, ErrNoIngredients
}
if r.ImageURL == "" {
return Ranked{}, ErrNoImage
}
return Ranked{Candidate: c, Recipe: r, Score: harness.Similarity(r.Title, dish) + completeness(r)}, nil
}
// completeness is a quarter for each of ingredients, image, steps and total
// minutes the recipe has.
func completeness(r Recipe) float64 {
score := 0.0
for _, present := range []bool{len(r.Ingredients) > 0, r.ImageURL != "", len(r.Steps) > 0, r.TotalMinutes != nil} {
if present {
score += 0.25
}
}
return score
}
+255
View File
@@ -0,0 +1,255 @@
package recipes
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"testing"
)
var _ PageFetcher = (*Fetcher)(nil)
// fakeFetcher serves fixture pages by URL and records what was fetched.
type fakeFetcher struct {
pages map[string][]byte
mu sync.Mutex
fetched []string
}
func (f *fakeFetcher) Fetch(_ context.Context, rawURL string) (Page, error) {
f.mu.Lock()
f.fetched = append(f.fetched, rawURL)
f.mu.Unlock()
body, ok := f.pages[rawURL]
if !ok {
return Page{}, &StatusError{URL: rawURL, Status: 404}
}
return Page{URL: rawURL, ContentType: "text/html", Body: body}, nil
}
// recipePage returns a page whose Recipe has the given name and, as asked,
// ingredients, an image, steps and a total time.
func recipePage(name string, ingredients, image, steps, minutes bool) []byte {
ld := fmt.Sprintf(`{"@context":"https://schema.org","@type":"Recipe","name":%q`, name)
if ingredients {
ld += `,"recipeIngredient":["1 onion","2 cloves garlic"]`
}
if image {
ld += `,"image":"https://img.example.com/dish.jpg"`
}
if steps {
ld += `,"recipeInstructions":["Chop.","Cook."]`
}
if minutes {
ld += `,"totalTime":"PT40M"`
}
return page("", ld+"}")
}
func TestRankOrder(t *testing.T) {
f := &fakeFetcher{pages: map[string][]byte{
"https://a.example/thai": recipePage("Thai Green Curry", true, true, true, true),
"https://b.example/noimg": recipePage("Chicken Curry", true, false, true, true),
"https://c.example/easy": recipePage("Easy Chicken Curry", true, true, true, false),
"https://d.example/noingr": recipePage("Chicken Curry", false, true, true, true),
"https://e.example/best": recipePage("Chicken Curry", true, true, true, true),
"https://f.example/toodeep": recipePage("Chicken Curry", true, true, true, true),
}}
cands := []Candidate{
{URL: "https://a.example/thai", Title: "Thai green curry"},
{URL: "https://b.example/noimg"},
{URL: "https://c.example/easy"},
{URL: "https://d.example/noingr"},
{URL: "https://e.example/best"},
{URL: "https://f.example/toodeep"},
}
got, err := Rank(context.Background(), f, "Chicken curry", cands)
if !errors.Is(err, ErrCandidatesUnranked) || !errors.Is(err, ErrNoImage) || !errors.Is(err, ErrNoIngredients) {
t.Errorf("err = %v, want ErrCandidatesUnranked with ErrNoImage and ErrNoIngredients", err)
}
if errors.Is(err, ErrNoUsableRecipe) {
t.Errorf("err = %v, want not ErrNoUsableRecipe when some are ranked", err)
}
want := []struct {
url string
score float64
}{
// Same words, every field: 1 + 1.
{"https://e.example/best", 2},
// "Easy" is a filler word, so the title matches; no total time.
{"https://c.example/easy", 1.75},
// Shares one of four words: 0.25 + 1.
{"https://a.example/thai", 1.25},
}
if len(got) != len(want) {
t.Fatalf("got %d ranked, want %d: %+v", len(got), len(want), got)
}
for i, w := range want {
if got[i].Candidate.URL != w.url || got[i].Score != w.score {
t.Errorf("rank %d = %s (%.2f), want %s (%.2f)", i, got[i].Candidate.URL, got[i].Score, w.url, w.score)
}
}
if got[2].Recipe.Title != "Thai Green Curry" || got[2].Candidate.Title != "Thai green curry" {
t.Errorf("ranked entry lost its recipe or candidate: %+v", got[2])
}
for _, u := range f.fetched {
if u == "https://f.example/toodeep" {
t.Errorf("fetched a candidate past RankDepth")
}
}
if len(f.fetched) != RankDepth {
t.Errorf("fetched %d pages, want %d", len(f.fetched), RankDepth)
}
}
func TestRankTiesKeepSearchOrder(t *testing.T) {
f := &fakeFetcher{pages: map[string][]byte{
"https://a.example/1": recipePage("Beef Stew", true, true, true, true),
"https://b.example/2": recipePage("Beef Stew", true, true, true, true),
"https://c.example/3": recipePage("Beef Stew", true, true, true, true),
}}
cands := []Candidate{{URL: "https://a.example/1"}, {URL: "https://b.example/2"}, {URL: "https://c.example/3"}}
got, err := Rank(context.Background(), f, "beef stew", cands)
if err != nil {
t.Fatal(err)
}
for i, c := range cands {
if got[i].Candidate.URL != c.URL {
t.Errorf("rank %d = %s, want %s", i, got[i].Candidate.URL, c.URL)
}
}
}
// TestRankReportsFetchFailure checks a failed fetch is reported with the
// candidates that were ranked, never swallowed.
func TestRankReportsFetchFailure(t *testing.T) {
f := &fakeFetcher{pages: map[string][]byte{
"https://a.example/1": recipePage("Beef Stew", true, true, true, true),
}}
cands := []Candidate{{URL: "https://a.example/1"}, {URL: "https://b.example/missing"}}
got, err := Rank(context.Background(), f, "beef stew", cands)
if len(got) != 1 || got[0].Candidate.URL != "https://a.example/1" {
t.Errorf("ranked = %+v, want only https://a.example/1", got)
}
if !errors.Is(err, ErrCandidatesUnranked) {
t.Errorf("err = %v, want ErrCandidatesUnranked", err)
}
var se *StatusError
if !errors.As(err, &se) || se.Status != 404 || se.URL != "https://b.example/missing" {
t.Errorf("err = %v, want the 404 StatusError for https://b.example/missing", err)
}
if err != nil && !strings.Contains(err.Error(), "https://b.example/missing") {
t.Errorf("err = %v, want it to name the failed candidate", err)
}
}
func TestRankAllRejected(t *testing.T) {
f := &fakeFetcher{pages: map[string][]byte{
"https://a.example/noimg": recipePage("Chicken Curry", true, false, true, true),
"https://b.example/noingr": recipePage("Chicken Curry", false, true, true, true),
"https://c.example/blog": page(""),
}}
cands := []Candidate{
{URL: "https://a.example/noimg"},
{URL: "https://b.example/noingr"},
{URL: "https://c.example/blog"},
{URL: "https://d.example/missing"},
}
_, err := Rank(context.Background(), f, "chicken curry", cands)
for _, want := range []error{ErrNoUsableRecipe, ErrNoImage, ErrNoIngredients, ErrNoRecipe} {
if !errors.Is(err, want) {
t.Errorf("err = %v, want it to wrap %v", err, want)
}
}
var se *StatusError
if !errors.As(err, &se) || se.Status != 404 {
t.Errorf("err = %v, want the 404 StatusError", err)
}
}
func TestRankNoCandidates(t *testing.T) {
if _, err := Rank(context.Background(), &fakeFetcher{}, "chicken curry", nil); !errors.Is(err, ErrNoUsableRecipe) {
t.Errorf("err = %v, want ErrNoUsableRecipe", err)
}
}
func TestRankContextCancelled(t *testing.T) {
f := &fakeFetcher{pages: map[string][]byte{"https://a.example/1": recipePage("Beef Stew", true, true, true, true)}}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := Rank(ctx, f, "beef stew", []Candidate{{URL: "https://a.example/1"}}); !errors.Is(err, context.Canceled) {
t.Errorf("err = %v, want context.Canceled", err)
}
}
// TestRankOverHTTP ranks fixture candidates served over HTTPS through the
// SSRF-safe Fetcher: every host, a subdomain the test server's certificate
// covers, resolves to a public test address the test
// dialer routes to one httptest server, which serves pages by host and path.
func TestRankOverHTTP(t *testing.T) {
t.Parallel()
pages := map[string][]byte{
"best.example.com/curry": recipePage("Chicken Curry", true, true, true, true),
"easy.example.com/curry": recipePage("Easy Chicken Curry", true, true, true, false),
"thai.example.com/curry": recipePage("Thai Green Curry", true, true, true, true),
"noimg.example.com/curry": recipePage("Chicken Curry", true, false, true, true),
}
env := newTestEnv(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, ok := pages[r.Host+r.URL.Path]
if !ok {
http.Error(w, "broken", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html")
w.Write(body)
}), map[string][][]string{
"best.example.com": {{publicIP}},
"easy.example.com": {{publicIP}},
"thai.example.com": {{publicIP}},
"noimg.example.com": {{publicIP}},
"broken.example.com": {{publicIP}},
})
cands := []Candidate{
{URL: "https://thai.example.com/curry"},
{URL: "https://broken.example.com/curry"},
{URL: "https://easy.example.com/curry"},
{URL: "https://noimg.example.com/curry"},
{URL: "https://best.example.com/curry"},
}
got, err := Rank(t.Context(), env.f, "Chicken curry", cands)
want := []struct {
url string
score float64
}{
{"https://best.example.com/curry", 2},
{"https://easy.example.com/curry", 1.75},
{"https://thai.example.com/curry", 1.25},
}
if len(got) != len(want) {
t.Fatalf("got %d ranked, want %d: %+v (err %v)", len(got), len(want), got, err)
}
for i, w := range want {
if got[i].Candidate.URL != w.url || got[i].Score != w.score {
t.Errorf("rank %d = %s (%.2f), want %s (%.2f)", i, got[i].Candidate.URL, got[i].Score, w.url, w.score)
}
}
if got[0].Recipe.Title != "Chicken Curry" || got[0].Recipe.ImageURL != "https://img.example.com/dish.jpg" {
t.Errorf("top recipe = %+v", got[0].Recipe)
}
if !errors.Is(err, ErrCandidatesUnranked) || !errors.Is(err, ErrNoImage) {
t.Errorf("err = %v, want ErrCandidatesUnranked with ErrNoImage", err)
}
var se *StatusError
if !errors.As(err, &se) || se.Status != http.StatusInternalServerError {
t.Errorf("err = %v, want the 500 StatusError for broken.example.com", err)
}
if n := len(env.dials()); n != len(cands) {
t.Errorf("dialed %d times, want %d", n, len(cands))
}
}
+314
View File
@@ -0,0 +1,314 @@
package recipes
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/secretbox"
)
const (
// MaxSearchResults is the most candidates one search may ask for; it is
// Brave's largest page.
MaxSearchResults = 20
// SearchTimeout bounds one search request, reading the body included.
SearchTimeout = 10 * time.Second
// MaxSearchBodyBytes caps a search response body. A larger one is an
// error, never truncated.
MaxSearchBodyBytes = 1 << 20
// BraveEndpoint is the Brave Web Search API.
BraveEndpoint = "https://api.search.brave.com/res/v1/web/search"
)
var (
// ErrSearchRefused is returned for HTTP 401 or 403: Brave rejected the
// API key, or SearXNG does not allow format=json.
ErrSearchRefused = errors.New("recipes: search provider refused the request")
// ErrSearchRateLimited is returned for HTTP 429.
ErrSearchRateLimited = errors.New("recipes: search provider rate limit reached")
// ErrSearchResponse is returned for a response body that is not the
// provider's JSON shape — SearXNG without a results array, Brave without
// "type": "search" or with a web object lacking a results array — is
// over MaxSearchBodyBytes, or holds a result without an absolute http(s)
// URL.
ErrSearchResponse = errors.New("recipes: malformed search response")
)
// SearchStatusError is returned for any other non-2xx search response,
// redirects included: a search endpoint is called directly and never
// followed elsewhere.
type SearchStatusError struct {
Provider string
Status int
}
func (e *SearchStatusError) Error() string {
return fmt.Sprintf("recipes: %s search: HTTP %d", e.Provider, e.Status)
}
// Candidate is one search result: a page that may hold a recipe.
type Candidate struct {
URL string
Title string
Snippet string
}
// SearchProvider finds recipe pages on the web.
type SearchProvider interface {
// Search returns up to n candidates for query, best first. The query
// gets " recipe" appended unless it already has the word. n must be
// 1 to MaxSearchResults.
Search(ctx context.Context, query string, n int) ([]Candidate, error)
}
// NewSearchProvider returns the provider s names. SearXNG is called at
// searxngURL, the configured MEALQUEST_SEARXNG_URL; Brave is called with the
// API key in s.ApiKeyCt, decrypted by box.
func NewSearchProvider(s db.RecipeSearchSetting, box *secretbox.Box, searxngURL string) (SearchProvider, error) {
switch s.Provider {
case "searxng":
return NewSearXNG(searxngURL)
case "brave":
if len(s.ApiKeyCt) == 0 {
return nil, errors.New("recipes: Brave search has no API key")
}
key, err := box.Decrypt(s.ApiKeyCt)
if err != nil {
return nil, fmt.Errorf("recipes: Brave API key: %w", err)
}
return NewBrave(string(key))
default:
return nil, fmt.Errorf("recipes: unknown search provider %q", s.Provider)
}
}
// SearXNG searches through a SearXNG instance. Its base URL is a configured
// endpoint and may be on a private network, so it is called directly rather
// than through the public-only Fetcher.
type SearXNG struct {
endpoint *url.URL
client *http.Client
}
// NewSearXNG returns a SearXNG provider for the instance at baseURL, an
// absolute http:// or https:// URL without credentials, query or fragment.
func NewSearXNG(baseURL string) (*SearXNG, error) {
if baseURL == "" {
return nil, errors.New("recipes: SearXNG URL is not set")
}
u, err := url.Parse(baseURL)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return nil, errors.New("recipes: SearXNG URL must be an absolute http:// or https:// URL")
}
if u.User != nil || u.RawQuery != "" || u.Fragment != "" {
return nil, errors.New("recipes: SearXNG URL must not have credentials, a query or a fragment")
}
return &SearXNG{endpoint: u.JoinPath("search"), client: newSearchClient()}, nil
}
// Search implements SearchProvider with
// {base}/search?q=…&format=json&categories=general.
func (s *SearXNG) Search(ctx context.Context, query string, n int) ([]Candidate, error) {
q, err := searchQuery(query, n)
if err != nil {
return nil, err
}
u := *s.endpoint
u.RawQuery = url.Values{"q": {q}, "format": {"json"}, "categories": {"general"}}.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
if err != nil {
return nil, fmt.Errorf("recipes: build SearXNG request: %w", err)
}
// Results is a pointer so a body without the array is told apart from
// an empty search.
var body struct {
Results *[]struct {
URL string `json:"url"`
Title string `json:"title"`
Content string `json:"content"`
} `json:"results"`
}
if err := doSearch(s.client, req, "SearXNG", &body); err != nil {
return nil, err
}
if body.Results == nil {
return nil, fmt.Errorf("%w: SearXNG body has no results array", ErrSearchResponse)
}
var out []Candidate
for _, r := range *body.Results {
out = append(out, Candidate{URL: r.URL, Title: r.Title, Snippet: r.Content})
}
return candidates("SearXNG", out, n)
}
// Brave searches through the Brave Web Search API.
type Brave struct {
key string
endpoint string
client *http.Client
}
// NewBrave returns a Brave provider that authenticates with key, the
// decrypted API key.
func NewBrave(key string) (*Brave, error) {
if key == "" {
return nil, errors.New("recipes: Brave API key is empty")
}
for i := 0; i < len(key); i++ {
if key[i] < 0x21 || key[i] > 0x7e {
// The key is never echoed.
return nil, errors.New("recipes: Brave API key has a character not allowed in a header")
}
}
return &Brave{key: key, endpoint: BraveEndpoint, client: newSearchClient()}, nil
}
// Search implements SearchProvider with GET BraveEndpoint?q=…&count=n.
func (b *Brave) Search(ctx context.Context, query string, n int) ([]Candidate, error) {
q, err := searchQuery(query, n)
if err != nil {
return nil, err
}
u := b.endpoint + "?" + url.Values{"q": {q}, "count": {strconv.Itoa(n)}}.Encode()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, fmt.Errorf("recipes: build Brave request: %w", err)
}
req.Header.Set("X-Subscription-Token", b.key)
// Brave leaves out "web" when nothing matched, so a search response is
// recognised by its "type"; a web object present must hold the results
// array.
var body struct {
Type string `json:"type"`
Web *struct {
Results *[]struct {
URL string `json:"url"`
Title string `json:"title"`
Description string `json:"description"`
} `json:"results"`
} `json:"web"`
}
if err := doSearch(b.client, req, "Brave", &body); err != nil {
return nil, err
}
if body.Type != "search" {
return nil, fmt.Errorf("%w: Brave body type is %q, not \"search\"", ErrSearchResponse, body.Type)
}
if body.Web == nil {
return nil, nil
}
if body.Web.Results == nil {
return nil, fmt.Errorf("%w: Brave web object has no results array", ErrSearchResponse)
}
var out []Candidate
for _, r := range *body.Web.Results {
out = append(out, Candidate{URL: r.URL, Title: r.Title, Snippet: r.Description})
}
return candidates("Brave", out, n)
}
func newSearchClient() *http.Client {
return &http.Client{
Timeout: SearchTimeout,
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
}
}
// searchQuery validates n and returns query with " recipe" appended unless
// one of its words already is "recipe", in any case.
func searchQuery(query string, n int) (string, error) {
if n < 1 || n > MaxSearchResults {
return "", fmt.Errorf("recipes: search result count %d is outside 1-%d", n, MaxSearchResults)
}
q := strings.TrimSpace(query)
if q == "" {
return "", errors.New("recipes: search query is empty")
}
for _, w := range strings.Fields(q) {
if strings.EqualFold(w, "recipe") {
return q, nil
}
}
return q + " recipe", nil
}
// doSearch sends req and decodes a 2xx JSON body into v, mapping failures to
// the search errors.
func doSearch(client *http.Client, req *http.Request, provider string, v any) error {
req.Header.Set("Accept", "application/json")
req.Header.Set("User-Agent", UserAgent)
resp, err := client.Do(req)
if err != nil {
if ctxErr := req.Context().Err(); ctxErr != nil {
return ctxErr
}
return fmt.Errorf("recipes: %s search: %w", provider, err)
}
defer resp.Body.Close()
switch {
case resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden:
return fmt.Errorf("%w: %s HTTP %d", ErrSearchRefused, provider, resp.StatusCode)
case resp.StatusCode == http.StatusTooManyRequests:
return fmt.Errorf("%w: %s", ErrSearchRateLimited, provider)
case resp.StatusCode < 200 || resp.StatusCode > 299:
return &SearchStatusError{Provider: provider, Status: resp.StatusCode}
}
data, err := io.ReadAll(io.LimitReader(resp.Body, MaxSearchBodyBytes+1))
if err != nil {
return fmt.Errorf("recipes: %s search: read body: %w", provider, err)
}
if len(data) > MaxSearchBodyBytes {
return fmt.Errorf("%w: %s body is over %d bytes", ErrSearchResponse, provider, MaxSearchBodyBytes)
}
if err := json.Unmarshal(data, v); err != nil {
return fmt.Errorf("%w: %s: %v", ErrSearchResponse, provider, err)
}
return nil
}
// candidates cleans results — HTML tags and entities, which both providers
// leave in titles and snippets, are removed — checks every URL is absolute
// http(s) and keeps the first n.
func candidates(provider string, results []Candidate, n int) ([]Candidate, error) {
if len(results) > n {
results = results[:n]
}
out := make([]Candidate, 0, len(results))
for i, r := range results {
u, err := url.Parse(r.URL)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return nil, fmt.Errorf("%w: %s result %d has no absolute http(s) URL", ErrSearchResponse, provider, i)
}
out = append(out, Candidate{URL: r.URL, Title: stripTags(r.Title), Snippet: stripTags(r.Snippet)})
}
return out, nil
}
// stripTags drops HTML tags such as Brave's <strong> highlights, then
// decodes entities and trims space.
func stripTags(s string) string {
var b strings.Builder
inTag := false
for _, r := range s {
switch {
case r == '<':
inTag = true
case r == '>' && inTag:
inTag = false
case !inTag:
b.WriteRune(r)
}
}
return cleanText(b.String())
}
+294
View File
@@ -0,0 +1,294 @@
package recipes
import (
"context"
"crypto/rand"
"encoding/base64"
"errors"
"net/http"
"net/http/httptest"
"reflect"
"strings"
"testing"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/secretbox"
)
const searxngBody = `{
"query": "chicken curry recipe",
"results": [
{"url": "https://www.bbcgoodfood.com/recipes/easy-chicken-curry", "title": "Easy chicken curry recipe | BBC Good Food", "content": "Make our easy chicken curry &amp; rice.", "engine": "duckduckgo"},
{"url": "https://www.recipetineats.com/chicken-curry/", "title": "Chicken Curry", "content": "A proper curry."},
{"url": "https://example.com/third", "title": "Third", "content": ""}
],
"number_of_results": 0
}`
const braveBody = `{
"type": "search",
"web": {
"type": "search",
"results": [
{"url": "https://www.bbcgoodfood.com/recipes/easy-chicken-curry", "title": "Easy <strong>chicken curry</strong> recipe", "description": "Make our <strong>easy</strong> chicken curry &amp; rice.", "language": "en"},
{"url": "https://www.recipetineats.com/chicken-curry/", "title": "Chicken Curry", "description": "A proper curry."}
]
}
}`
func newSearXNGServer(t *testing.T, status int, body string, got *http.Request) *SearXNG {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got != nil {
*got = *r.Clone(context.Background())
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
s, err := NewSearXNG(srv.URL + "/searx")
if err != nil {
t.Fatal(err)
}
return s
}
func newBraveServer(t *testing.T, key string, status int, body string, got *http.Request) *Brave {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got != nil {
*got = *r.Clone(context.Background())
}
w.WriteHeader(status)
w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
b, err := NewBrave(key)
if err != nil {
t.Fatal(err)
}
b.endpoint = srv.URL + "/res/v1/web/search"
return b
}
func TestSearXNGSearch(t *testing.T) {
var req http.Request
s := newSearXNGServer(t, http.StatusOK, searxngBody, &req)
got, err := s.Search(context.Background(), "chicken curry", 2)
if err != nil {
t.Fatal(err)
}
if req.URL.Path != "/searx/search" {
t.Errorf("path = %q, want /searx/search", req.URL.Path)
}
q := req.URL.Query()
if q.Get("q") != "chicken curry recipe" || q.Get("format") != "json" || q.Get("categories") != "general" {
t.Errorf("query = %v", q)
}
if req.Header.Get("User-Agent") != UserAgent {
t.Errorf("User-Agent = %q", req.Header.Get("User-Agent"))
}
want := []Candidate{
{URL: "https://www.bbcgoodfood.com/recipes/easy-chicken-curry", Title: "Easy chicken curry recipe | BBC Good Food", Snippet: "Make our easy chicken curry & rice."},
{URL: "https://www.recipetineats.com/chicken-curry/", Title: "Chicken Curry", Snippet: "A proper curry."},
}
if !reflect.DeepEqual(got, want) {
t.Errorf("candidates =\n%#v\nwant\n%#v", got, want)
}
}
func TestBraveSearch(t *testing.T) {
var req http.Request
b := newBraveServer(t, "brave-key-123", http.StatusOK, braveBody, &req)
got, err := b.Search(context.Background(), "Chicken curry RECIPE", 5)
if err != nil {
t.Fatal(err)
}
if req.Header.Get("X-Subscription-Token") != "brave-key-123" {
t.Errorf("X-Subscription-Token = %q", req.Header.Get("X-Subscription-Token"))
}
if req.Header.Get("Accept") != "application/json" {
t.Errorf("Accept = %q", req.Header.Get("Accept"))
}
q := req.URL.Query()
if q.Get("q") != "Chicken curry RECIPE" || q.Get("count") != "5" {
t.Errorf("query = %v", q)
}
want := []Candidate{
{URL: "https://www.bbcgoodfood.com/recipes/easy-chicken-curry", Title: "Easy chicken curry recipe", Snippet: "Make our easy chicken curry & rice."},
{URL: "https://www.recipetineats.com/chicken-curry/", Title: "Chicken Curry", Snippet: "A proper curry."},
}
if !reflect.DeepEqual(got, want) {
t.Errorf("candidates =\n%#v\nwant\n%#v", got, want)
}
}
func TestSearchQuery(t *testing.T) {
for _, tc := range []struct{ in, want string }{
{"chicken curry", "chicken curry recipe"},
{" chicken curry ", "chicken curry recipe"},
{"chicken curry recipe", "chicken curry recipe"},
{"Recipe for chicken curry", "Recipe for chicken curry"},
{"recipes for curry", "recipes for curry recipe"},
} {
got, err := searchQuery(tc.in, 3)
if err != nil || got != tc.want {
t.Errorf("searchQuery(%q) = %q, %v; want %q", tc.in, got, err, tc.want)
}
}
for _, tc := range []struct {
q string
n int
}{{"", 3}, {" ", 3}, {"curry", 0}, {"curry", MaxSearchResults + 1}} {
if _, err := searchQuery(tc.q, tc.n); err == nil {
t.Errorf("searchQuery(%q, %d) succeeded, want error", tc.q, tc.n)
}
}
}
func TestSearchErrors(t *testing.T) {
tests := []struct {
name string
status int
body string
check func(error) bool
}{
{"unauthorized", http.StatusUnauthorized, `{}`, func(err error) bool { return errors.Is(err, ErrSearchRefused) }},
{"forbidden", http.StatusForbidden, `{}`, func(err error) bool { return errors.Is(err, ErrSearchRefused) }},
{"rate limited", http.StatusTooManyRequests, `{}`, func(err error) bool { return errors.Is(err, ErrSearchRateLimited) }},
{"server error", http.StatusBadGateway, `{}`, func(err error) bool {
var se *SearchStatusError
return errors.As(err, &se) && se.Status == http.StatusBadGateway
}},
{"redirect not followed", http.StatusFound, ``, func(err error) bool {
var se *SearchStatusError
return errors.As(err, &se) && se.Status == http.StatusFound
}},
{"not JSON", http.StatusOK, `<html>`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"result without URL", http.StatusOK, `{"type":"search","results":[{"title":"x"}],"web":{"results":[{"title":"x"}]}}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"relative URL", http.StatusOK, `{"type":"search","results":[{"url":"/x"}],"web":{"results":[{"url":"/x"}]}}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"empty object", http.StatusOK, `{}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"null results", http.StatusOK, `{"type":"search","results":null,"web":{"results":null}}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"results not an array", http.StatusOK, `{"type":"search","results":{},"web":{"results":{}}}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
{"body too large", http.StatusOK, `{"results":[],"pad":"` + strings.Repeat("x", MaxSearchBodyBytes) + `"}`, func(err error) bool { return errors.Is(err, ErrSearchResponse) }},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
providers := map[string]SearchProvider{
"SearXNG": newSearXNGServer(t, tc.status, tc.body, nil),
"Brave": newBraveServer(t, "secret-key", tc.status, tc.body, nil),
}
for name, p := range providers {
_, err := p.Search(context.Background(), "curry", 3)
if err == nil || !tc.check(err) {
t.Errorf("%s: err = %v", name, err)
}
if err != nil && strings.Contains(err.Error(), "secret-key") {
t.Errorf("%s: error leaks the API key: %v", name, err)
}
}
})
}
}
// TestSearchNoMatches checks a well-formed response with nothing found is an
// empty search, not an error.
func TestSearchNoMatches(t *testing.T) {
providers := map[string]SearchProvider{
"SearXNG": newSearXNGServer(t, http.StatusOK, `{"query":"x recipe","results":[]}`, nil),
"Brave without web": newBraveServer(t, "secret-key", http.StatusOK, `{"type":"search","query":{"original":"x recipe"}}`, nil),
"Brave empty web": newBraveServer(t, "secret-key", http.StatusOK, `{"type":"search","web":{"type":"search","results":[]}}`, nil),
}
for name, p := range providers {
got, err := p.Search(context.Background(), "x", 3)
if err != nil || len(got) != 0 {
t.Errorf("%s: got %v, %v; want no candidates and no error", name, got, err)
}
}
}
func TestSearchContextCancelled(t *testing.T) {
s := newSearXNGServer(t, http.StatusOK, searxngBody, nil)
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := s.Search(ctx, "curry", 3); !errors.Is(err, context.Canceled) {
t.Errorf("err = %v, want context.Canceled", err)
}
}
func TestSearchUnreachable(t *testing.T) {
srv := httptest.NewServer(http.NotFoundHandler())
url := srv.URL
srv.Close()
s, err := NewSearXNG(url)
if err != nil {
t.Fatal(err)
}
_, err = s.Search(context.Background(), "curry", 3)
if err == nil || !strings.Contains(err.Error(), "SearXNG search") {
t.Errorf("err = %v", err)
}
}
func TestNewSearXNGRejects(t *testing.T) {
for _, u := range []string{"", "searxng:8080", "ftp://searxng", "http://", "http://u:p@searxng", "http://searxng?x=1", "http://searxng#f"} {
if _, err := NewSearXNG(u); err == nil {
t.Errorf("NewSearXNG(%q) succeeded, want error", u)
}
}
if _, err := NewSearXNG("http://searxng:8080"); err != nil {
t.Errorf("internal URL rejected: %v", err)
}
}
func TestNewBraveRejects(t *testing.T) {
for _, k := range []string{"", "with space", "line\nbreak"} {
if _, err := NewBrave(k); err == nil {
t.Errorf("NewBrave(%q) succeeded, want error", k)
}
}
}
func TestNewSearchProvider(t *testing.T) {
raw := make([]byte, secretbox.KeySize)
rand.Read(raw)
box, err := secretbox.New(base64.StdEncoding.EncodeToString(raw))
if err != nil {
t.Fatal(err)
}
ct, err := box.Encrypt([]byte("brave-key"))
if err != nil {
t.Fatal(err)
}
p, err := NewSearchProvider(db.RecipeSearchSetting{Provider: "brave", ApiKeyCt: ct}, box, "")
if err != nil {
t.Fatal(err)
}
if b, ok := p.(*Brave); !ok || b.key != "brave-key" || b.endpoint != BraveEndpoint {
t.Errorf("provider = %#v, want Brave with decrypted key", p)
}
p, err = NewSearchProvider(db.RecipeSearchSetting{Provider: "searxng"}, box, "http://searxng:8080")
if err != nil {
t.Fatal(err)
}
if s, ok := p.(*SearXNG); !ok || s.endpoint.String() != "http://searxng:8080/search" {
t.Errorf("provider = %#v, want SearXNG at http://searxng:8080/search", p)
}
for name, tc := range map[string]struct {
s db.RecipeSearchSetting
url string
}{
"brave without key": {db.RecipeSearchSetting{Provider: "brave"}, ""},
"brave bad key": {db.RecipeSearchSetting{Provider: "brave", ApiKeyCt: []byte("not a token at all, far too plain")}, ""},
"searxng without URL": {db.RecipeSearchSetting{Provider: "searxng"}, ""},
"unknown provider": {db.RecipeSearchSetting{Provider: "google"}, ""},
} {
if _, err := NewSearchProvider(tc.s, box, tc.url); err == nil {
t.Errorf("%s: succeeded, want error", name)
}
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 KiB

+1 -1
View File
@@ -83,7 +83,7 @@ func newLoginRouter(t *testing.T, issuer string) http.Handler {
t.Fatal(err)
}
r, err := NewRouter(logger, "dev", trusted, flow, sessions, newTestSettings(t),
newTestRunner(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
newTestRunner(t), newTestPhotos(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
if err != nil {
t.Fatal(err)
}
+75
View File
@@ -0,0 +1,75 @@
package server
import (
"context"
"errors"
"io/fs"
"log/slog"
"net/http"
"os"
"strconv"
"strings"
"github.com/go-chi/chi/v5"
)
// photoCacheControl lets the browser keep a photo for a year without
// revalidating: a photo's files never change once stored, and a new image
// gets a new id.
const photoCacheControl = "private, max-age=31536000, immutable"
// PhotoFiles opens a cached photo's file at one of its stored sizes. An
// unknown photo or size is an error matching fs.ErrNotExist.
type PhotoFiles interface {
OpenPhoto(ctx context.Context, id string, size int) (*os.File, error)
}
// getPhoto answers GET /media/photos/{id}-{size}.jpg with the stored JPEG.
func getPhoto(logger *slog.Logger, photos PhotoFiles) http.HandlerFunc {
notFound := apiErrorHandler(logger, http.StatusNotFound, "not_found", "not found")
return func(w http.ResponseWriter, r *http.Request) {
id, size, ok := parsePhotoName(chi.URLParam(r, "name"))
if !ok {
notFound(w, r)
return
}
f, err := photos.OpenPhoto(r.Context(), id, size)
if errors.Is(err, fs.ErrNotExist) {
notFound(w, r)
return
}
if err != nil {
logger.ErrorContext(r.Context(), "open photo", "id", id, "size", size, "error", err)
writeError(w, r, logger, http.StatusInternalServerError, "internal", "internal error")
return
}
defer f.Close()
info, err := f.Stat()
if err != nil {
logger.ErrorContext(r.Context(), "stat photo", "id", id, "size", size, "error", err)
writeError(w, r, logger, http.StatusInternalServerError, "internal", "internal error")
return
}
w.Header().Set("Content-Type", "image/jpeg")
w.Header().Set("Cache-Control", photoCacheControl)
http.ServeContent(w, r, "", info.ModTime(), f)
}
}
// parsePhotoName splits "{id}-{size}.jpg" into its id and size, refusing
// a size not written in its canonical decimal form.
func parsePhotoName(name string) (id string, size int, ok bool) {
stem, ok := strings.CutSuffix(name, ".jpg")
if !ok {
return "", 0, false
}
i := strings.LastIndexByte(stem, '-')
if i < 0 {
return "", 0, false
}
size, err := strconv.Atoi(stem[i+1:])
if err != nil || strconv.Itoa(size) != stem[i+1:] {
return "", 0, false
}
return stem[:i], size, true
}
+138
View File
@@ -0,0 +1,138 @@
package server
import (
"io"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"testing/fstest"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/db"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/recipes"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
)
// newMediaRouter returns a router over a photo cache whose data directory
// holds both sizes of two photos, the id of the one with a photos row, and
// the id of the orphan without one. Each size's file holds its size as
// text.
func newMediaRouter(t *testing.T) (r http.Handler, id, orphan string) {
t.Helper()
dataDir := t.TempDir()
st := newTestStore(t)
photos, err := recipes.NewPhotoCache(recipes.NewFetcher(), st, dataDir)
if err != nil {
t.Fatal(err)
}
id, orphan = store.NewID(), store.NewID()
for _, stem := range []string{id, orphan} {
for _, size := range []string{"400", "1200"} {
p := filepath.Join(dataDir, "media", "photos", stem+"-"+size+".jpg")
if err := os.WriteFile(p, []byte("jpeg "+size), 0o600); err != nil {
t.Fatal(err)
}
}
}
now := store.Now().String()
if _, err := db.New(st.Writer).CreatePhoto(t.Context(), db.CreatePhotoParams{
ID: id,
SourceUrl: "https://example.com/a.jpg",
LocalPath: "media/photos/" + id + "-1200.jpg",
Width: 1600,
Height: 1200,
Sha256: "sha",
FetchedAt: now,
CreatedAt: now,
UpdatedAt: now,
}); err != nil {
t.Fatal(err)
}
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
r, err = NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, newTestSettings(t), newTestRunner(t),
photos, fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
if err != nil {
t.Fatal(err)
}
return r, id, orphan
}
func TestGetPhotoServesEachSize(t *testing.T) {
t.Parallel()
router, id, _ := newMediaRouter(t)
for _, size := range []string{"400", "1200"} {
req := httptest.NewRequest(http.MethodGet, "/media/photos/"+id+"-"+size+".jpg", nil)
req.AddCookie(validSessionCookie)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("size %s: status = %d, want 200", size, rec.Code)
}
if got := rec.Body.String(); got != "jpeg "+size {
t.Errorf("size %s: body = %q", size, got)
}
if got := rec.Header().Get("Content-Type"); got != "image/jpeg" {
t.Errorf("size %s: Content-Type = %q, want image/jpeg", size, got)
}
if got := rec.Header().Get("Cache-Control"); got != "private, max-age=31536000, immutable" {
t.Errorf("size %s: Cache-Control = %q", size, got)
}
}
}
func TestGetPhotoRequiresSession(t *testing.T) {
t.Parallel()
router, id, _ := newMediaRouter(t)
req := httptest.NewRequest(http.MethodGet, "/media/photos/"+id+"-400.jpg", nil)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", rec.Code)
}
if got := rec.Body.String(); got != unauthenticatedJSON {
t.Errorf("body = %s, want %s", got, unauthenticatedJSON)
}
if got := rec.Header().Get("Cache-Control"); got != "" {
t.Errorf("Cache-Control = %q on a refused request", got)
}
}
func TestGetPhotoUnknownNotFound(t *testing.T) {
t.Parallel()
router, id, orphan := newMediaRouter(t)
for _, name := range []string{
store.NewID() + "-400.jpg", // no such photo
orphan + "-400.jpg", // file stored, no photos row
id + "-800.jpg", // size not stored
orphan + "-1200.jpg",
id + "-0400.jpg",
id + "-+400.jpg",
id + "-400.png",
id + ".jpg",
id + "-400",
"..%2F" + id + "-400.jpg",
"not-an-id-400.jpg",
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
req := httptest.NewRequest(http.MethodGet, "/media/photos/"+name, nil)
req.AddCookie(validSessionCookie)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
if rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404", rec.Code)
}
if got := rec.Body.String(); got != notFoundJSON {
t.Errorf("body = %s, want %s", got, notFoundJSON)
}
if got := rec.Header().Get("Cache-Control"); got != "" {
t.Errorf("Cache-Control = %q on a 404", got)
}
})
}
}
+1 -1
View File
@@ -45,7 +45,7 @@ func newOriginRouter(t *testing.T, logs io.Writer) http.Handler {
}
logger := slog.New(forwarded.LogHandler(slog.NewJSONHandler(logs, nil)))
r, err := NewRouter(logger, "dev", trusted, probeAuth{}, stubSessions{}, newTestSettings(t),
newTestRunner(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
newTestRunner(t), newTestPhotos(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
if err != nil {
t.Fatal(err)
}
+7 -2
View File
@@ -30,16 +30,20 @@ type RouteMounter interface {
// only from a peer in trusted (see resolveOrigin). version is reported by
// GET /health; auth mounts the /auth routes; sessions guards /api/v1 and
// /media; settings serves /api/v1/settings/household; runner serves
// POST /api/v1/jobs and GET /api/v1/jobs/{id}; assets is the web build
// POST /api/v1/jobs and GET /api/v1/jobs/{id}; photos serves
// GET /media/photos/{id}-{size}.jpg; assets is the web build
// served for every other GET. /health, /auth and the web build are public.
// Requests to /api/v1 other than GET also need the session's CSRF token in CSRFHeader.
func NewRouter(logger *slog.Logger, version string, trusted forwarded.Trusted, auth RouteMounter, sessions SessionLookup, settings *household.SettingsService, runner *jobs.Runner, assets fs.FS) (http.Handler, error) {
func NewRouter(logger *slog.Logger, version string, trusted forwarded.Trusted, auth RouteMounter, sessions SessionLookup, settings *household.SettingsService, runner *jobs.Runner, photos PhotoFiles, assets fs.FS) (http.Handler, error) {
if settings == nil {
return nil, errors.New("server: nil household settings service")
}
if runner == nil {
return nil, errors.New("server: nil job runner")
}
if photos == nil {
return nil, errors.New("server: nil photo files")
}
notFound := apiErrorHandler(logger, http.StatusNotFound, "not_found", "not found")
methodNotAllowed := apiErrorHandler(logger, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
@@ -70,6 +74,7 @@ func NewRouter(logger *slog.Logger, version string, trusted forwarded.Trusted, a
apiV1.Patch("/settings/household", patchHouseholdSettings(logger, settings))
apiV1.Post("/jobs", postJob(logger, runner))
apiV1.Get("/jobs/{id}", getJob(logger, runner))
media.Get("/photos/{name}", getPhoto(logger, photos))
r.Route("/api", func(r chi.Router) {
r.MethodNotAllowed(methodNotAllowed)
r.NotFound(notFound)
+24 -3
View File
@@ -16,6 +16,7 @@ import (
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/household"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/jobs"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/recipes"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/internal/store"
"code.cubecraftlabs.com/Maniacs_Mansion/Meal-Quest/migrations"
)
@@ -38,7 +39,7 @@ func newTestRouterWith(t *testing.T, version string, settings *household.Setting
"index.html": {Data: []byte(fixtureIndex)},
"assets/index-AbC123.js": {Data: []byte(fixtureAsset)},
}
r, err := NewRouter(logger, version, nil, stubAuth{}, stubSessions{}, settings, runner, assets)
r, err := NewRouter(logger, version, nil, stubAuth{}, stubSessions{}, settings, runner, newTestPhotos(t), assets)
if err != nil {
t.Fatal(err)
}
@@ -112,11 +113,31 @@ func newTestRunner(t *testing.T) *jobs.Runner {
return runner
}
// newTestPhotos returns a photo cache over a migrated temp database and an
// empty temp data directory.
func newTestPhotos(t *testing.T) *recipes.PhotoCache {
t.Helper()
photos, err := recipes.NewPhotoCache(recipes.NewFetcher(), newTestStore(t), t.TempDir())
if err != nil {
t.Fatal(err)
}
return photos
}
func TestNewRouterRequiresPhotoFiles(t *testing.T) {
t.Parallel()
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
if _, err := NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, newTestSettings(t), newTestRunner(t), nil, fstest.MapFS{"index.html": {}}); err == nil {
t.Fatal("NewRouter accepted nil photo files")
}
}
func TestNewRouterRequiresSettings(t *testing.T) {
t.Parallel()
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
if _, err := NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, nil, newTestRunner(t), fstest.MapFS{"index.html": {}}); err == nil {
if _, err := NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, nil, newTestRunner(t), newTestPhotos(t), fstest.MapFS{"index.html": {}}); err == nil {
t.Fatal("NewRouter accepted a nil settings service")
}
}
@@ -125,7 +146,7 @@ func TestNewRouterRequiresJobRunner(t *testing.T) {
t.Parallel()
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
if _, err := NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, newTestSettings(t), nil, fstest.MapFS{"index.html": {}}); err == nil {
if _, err := NewRouter(logger, "dev", nil, stubAuth{}, stubSessions{}, newTestSettings(t), nil, newTestPhotos(t), fstest.MapFS{"index.html": {}}); err == nil {
t.Fatal("NewRouter accepted a nil job runner")
}
}
+1 -1
View File
@@ -216,7 +216,7 @@ func TestHouseholdSettingsWithRealSession(t *testing.T) {
}
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
router, err := NewRouter(logger, "dev", nil, stubAuth{}, sessions, household.NewSettingsService(st),
newTestRunner(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
newTestRunner(t), newTestPhotos(t), fstest.MapFS{"index.html": {Data: []byte(fixtureIndex)}})
if err != nil {
t.Fatal(err)
}
+4
View File
@@ -15,3 +15,7 @@ SELECT * FROM photos
WHERE sha256 = ?
ORDER BY id
LIMIT 1;
-- name: GetPhoto :one
SELECT * FROM photos
WHERE id = ?;