Record site-name provenance instead of inferring it from the name's shape
CI / ci (push) Successful in 8m11s
CI / ci (push) Successful in 8m11s
Recipe extraction no longer falls back to the URL's host: a stored recipes.site_name is now always a name the page gave (publisher or og:site_name), and null when it named none. dinners.SourceOf is the one documented rule for every fetched recipe's source: the page's name is shown as it stands; otherwise the site is named by the link's host only when the link itself is shown as free of secrets, else null. The recipe detail, idea results and the recipe-candidate offer endpoint all apply it; candidates no longer carry an extraction-time hostname fallback, so their site_name is nullable (frontend updated to match). Migration 0020 clears host-shaped site names (dotted host names, IP addresses) stored before the change, in recipes and in legacy idea job results, so already-stored rows read under the same rule. Maestro 966066e3-0e25-4b8b-9bb0-0694c47c9106 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+34
-37
@@ -58,14 +58,14 @@ type Ingredient struct {
|
||||
RawText string `json:"raw_text"`
|
||||
}
|
||||
|
||||
// Source is the site a recipe was fetched from. SiteName is the name the
|
||||
// page itself gives its site, and null when it gives none or the stored
|
||||
// name could have come from a link (see pageSiteName). URL is the page's
|
||||
// link only when it is verifiably free of secrets (see safeSourceURL), and
|
||||
// null otherwise: a link pasted or offered as a candidate may carry a
|
||||
// personal token, so a client links the credit only when URL is not null.
|
||||
// Source is set for every fetched recipe, so a client always credits it:
|
||||
// by SiteName, else by URL's host, else without naming the site.
|
||||
// Source is the site a recipe was fetched from, as SourceOf gives it.
|
||||
// SiteName is the name the page gives its site, else the host of URL when
|
||||
// URL is shown, else null. URL is the page's link only when it is
|
||||
// verifiably free of secrets (see safeSourceURL), and null otherwise: a
|
||||
// link pasted or offered as a candidate may carry a personal token, so a
|
||||
// client links the credit only when URL is not null. Source is set for
|
||||
// every fetched recipe, so a client always credits it: by SiteName, else
|
||||
// without naming the site.
|
||||
type Source struct {
|
||||
SiteName *string `json:"site_name"`
|
||||
URL *string `json:"url"`
|
||||
@@ -183,42 +183,39 @@ func readRecipe(ctx context.Context, q *db.Queries, recipeID string, d *Detail)
|
||||
}
|
||||
|
||||
// SourceOf is the Source of a recipe fetched from sourceURL whose stored
|
||||
// site name is siteName: the name only when the page gave it (see
|
||||
// pageSiteName) and the link only when it is free of secrets (see
|
||||
// safeSourceURL). It is the one way a fetched recipe's source reaches a
|
||||
// client. A sourceURL that does not parse is an error, whether or not a
|
||||
// site name is stored.
|
||||
// site name is siteName, the name its page gave (recipes.Recipe.SiteName),
|
||||
// null when the page named none. It is the one rule by which a fetched
|
||||
// recipe's source reaches a client — the recipe detail, an idea and a
|
||||
// recipe candidate offered for a dinner alike:
|
||||
//
|
||||
// - the link is shown only when it is free of secrets (see
|
||||
// safeSourceURL);
|
||||
// - a site name the page gave is shown as it stands, as the page
|
||||
// publishes it to everyone who reads it;
|
||||
// - otherwise the site is named by the link's host without a leading
|
||||
// "www.", and only when the link itself is shown, so the name tells
|
||||
// nothing the shown link does not; else it is null.
|
||||
//
|
||||
// A host is never stored as a site name, so a stored name is always the
|
||||
// page's: migration 0020 cleared the names stored before that, when a
|
||||
// page naming no site was stored under its host. A sourceURL that does not
|
||||
// parse is an error, whether or not a site name is stored.
|
||||
func SourceOf(siteName sql.NullString, sourceURL string) (Source, error) {
|
||||
link, err := url.Parse(sourceURL)
|
||||
if err != nil {
|
||||
// Not wrapped: url.Parse's error quotes the link.
|
||||
return Source{}, errors.New("source url does not parse")
|
||||
}
|
||||
return Source{SiteName: pageSiteName(siteName, link), URL: safeSourceURL(sourceURL)}, nil
|
||||
}
|
||||
|
||||
// pageSiteName is the stored site name of a recipe fetched from link
|
||||
// when it is one the page gave, else nil. A page that names no site is
|
||||
// stored under recipes.HostSiteName of the URL it was served from: the
|
||||
// pasted link's host, which may hold a personal token (a private
|
||||
// subdomain), or that of a page the link redirected to, which is not
|
||||
// stored. Nothing records which a stored name is, so any name that could
|
||||
// be a host — the link's own host, an IP address, or a dotted host name —
|
||||
// is treated as one and withheld. A site that names itself as a domain,
|
||||
// such as "Food.com", loses its name with them: a credit that does not
|
||||
// name the site is better than one that leaks a link.
|
||||
func pageSiteName(stored sql.NullString, link *url.URL) *string {
|
||||
if !stored.Valid {
|
||||
return nil
|
||||
src := Source{URL: safeSourceURL(sourceURL)}
|
||||
switch {
|
||||
case siteName.Valid && siteName.String != "":
|
||||
name := siteName.String
|
||||
src.SiteName = &name
|
||||
case src.URL != nil:
|
||||
host := strings.TrimPrefix(link.Hostname(), "www.")
|
||||
src.SiteName = &host
|
||||
}
|
||||
name := stored.String
|
||||
if strings.EqualFold(name, recipes.HostSiteName(link)) || net.ParseIP(name) != nil {
|
||||
return nil
|
||||
}
|
||||
if u, err := url.Parse("https://" + name + "/"); err == nil && u.Host == name && strings.Contains(name, ".") {
|
||||
return nil
|
||||
}
|
||||
return &name
|
||||
return src, nil
|
||||
}
|
||||
|
||||
// safeSourceURL is sourceURL when it is verifiably free of secrets — an
|
||||
|
||||
Reference in New Issue
Block a user