Eng brief · companion to the DPA / crawler doc

The Commerce catalog & its crawlers

We run two Meta catalogs holding the same ~16,336 homes: one is the correct real-estate home_listings catalog (what DPA LAB serves from), the other is a commerce catalog (Catalog_Homa_Commerce_LAL). They crawl our site in different ways — and the commerce one is worth understanding, because its model is pixel-driven, not feed-driven.

Companion to the DPA & catalog-crawler brief · sources at the bottom

TL;DR

  1. Our commerce catalog updates & validates items via the Meta Pixel + on-page microdata, not a scheduled feed. Per Meta: "your product information won't be updated in Commerce Manager until your pixel has fired at least once."
  2. That means every time the Meta Pixel fires on a product page — by a real user or a JS-rendering crawler — Meta may re-fetch that page and re-read its microdata. On our expensive /property/{id} route, that's another path that can trigger the RapidAPI/OpenAI reads.
  3. The crawler is the same one from the DPA doc — meta-externalads/1.1 — so the same fix applies: serve it (and the pixel-triggered re-fetch) cached/lightweight property data.
  4. The commerce catalog is a duplicate. No active LAB DPA campaign serves from it (they all use the home_listings catalog). If it exists only to seed lookalikes, it may be removable — which would shrink the crawl surface. Housing must not be advertised off a commerce catalog (policy).

1Two catalogs, two update models

home_listings — Catalog_Real_Estate

The real-estate vertical. Updated by a scheduled data feed (our worker's feed.csv). Meta re-ingests the feed on a schedule and re-crawls product URLs to validate. This is what all active LAB DPA campaigns serve from. (Feed refresh now paused.)

commerce — Catalog_Homa_Commerce_LAL

The generic-products vertical. Same 16,336 homes as products. Kept fresh primarily via the Meta Pixel + microdata on the page — updates are pixel-triggered, not feed-scheduled. Not used by any active LAB DPA campaign.

2How the commerce crawler actually works

A commerce catalog can be fed three ways: a data feed, a partner/manual upload, or the Meta Pixel + microdata path. The pixel path is the one with distinct crawl behavior:

Fields Meta reads from the microdata:

og:title            product:price:amount        productID / retailer_item_id
og:description      product:price:currency      item_group_id
og:image            product:availability        product:condition
og:url              (Schema.org / JSON-LD equivalents all accepted)
Why this matters for us: because updates are pixel-triggered, the commerce catalog effectively rides on top of page traffic. Every product-page load that fires the Meta Pixel — real user or JS-rendering crawler — is an opportunity for Meta to re-fetch and re-parse that page. On /property/{id}, which fires ViewContent and does live RapidAPI + OpenAI work, this is a second mechanism (besides the DPA feed crawl) that can drive expensive re-reads.

3Same crawler, same fix

The crawler doing the fetching is still Meta's ads crawler:

User-agentRole for the commerce catalog
meta-externalads/1.1 OURSFetches product URLs to validate / enrich catalog items for ads (both verticals)
facebookexternalhit/1.1Reads OG/microdata for link previews & sharing

So the fix is identical to the DPA brief — detect the crawler UA on the property route and serve cached/lightweight data (title, price, availability, image, OG/microdata tags) without the live RapidAPI + OpenAI calls. The microdata is exactly what Meta wants anyway, and it's cheap to serve statically:

if (uaIsMetaCrawler(req)) {
  return renderCachedProperty(id);   // OG + product microdata from cache; no live reads
}
Don't strip the pixel or microdata to "stop the crawl." The commerce catalog depends on the pixel firing + microdata being present to stay accurate. Removing them breaks the catalog; caching the crawler's response does not.

4Recommendations specific to the commerce catalog

5Sources (Meta docs)

Microdata Tags for Catalogs — OG/Schema.org/JSON-LD, pixel-triggered updatesdevelopers.facebook.com/docs/marketing-api/catalog/guides/microdata-tags Ways to Add & Manage Products in Your Catalog (feed / pixel / crawl data sources)facebook.com/business/help/384041892421495 Connect a Meta Pixel to your catalogue in Commerce Managerfacebook.com/business/help/1044262445604547 Meta web crawlers — user-agents & robots.txt controldevelopers.facebook.com/docs/sharing/webmasters/web-crawlers
← back to the DPA & catalog-crawler brief