# Phase 7 — Customer Insights (Journeys and Data)

> **Verification status.** "Finding the install surface" and "Installing" — including both
> products' real timings — were executed and verified. The ingestion / unification steps in
> the second half were **planned but not yet executed** at the time of writing. Treat those
> as a well-researched plan, not a validated runbook, and correct this file once you have
> done it.

Customer Insights ships as two separate products:

| Product | What it is | Formerly |
|---|---|---|
| **CI – Journeys** | Marketing automation: segments, journeys, emails, forms, events | Dynamics 365 Marketing |
| **CI – Data** | Customer Data Platform: ingest, unify, measures, segments | Dynamics 365 Customer Insights |

Neither is installed by default in a Caldova tenant.

---

## Checking what is installed

> ⚠️ **Read this before you tell anyone a product is missing.** There are three surfaces here and
> **all three can report "not installed" while the product is actually one click from working.**

**1. The Dataverse solution list is not evidence.** A fresh environment often contains
`DynamicsMKT_SendOptimization` ("Dynamics Marketing Send Optimization"), a small supporting
solution — **not** the app. Its presence does not mean Journeys is installed.

**2. The admin-center provisioning grid is not final either.** It shows per-environment
`Installed` / `Not Installed`, but that reflects **admin-driven provisioning only**. It can show
`Not Installed` for a tenant that is fully entitled and needs nothing more than a self-service
first run.

**3. ✅ The authoritative check is the product's own portal.** Navigate there as the tenant admin:

| Product | Portal |
|---|---|
| CI – Data | **`https://home.ci.ai.dynamics.com`** |
| CI – Journeys | the `Customer Insights - Journeys` app in the org (`/main.aspx?pagetype=apps`) |

### The self-service first run IS the install — do not wait it out

**Field report (another operator, 2026-08-26):** this skill told them CI – Data and CI – Journeys
were not installed. Both were actually available. **All they had to do was open
`home.ci.ai.dynamics.com` and select their region.** No admin-center install, no ticket, no wait.

If you land on a first-run screen asking for **country/region** (and possibly consent), that screen
is **the install step**. Fill it in.

> **Correction.** An earlier version of this file called that screen "transient provisioning UI"
> and advised waiting for it to disappear. **That was wrong, and it is what produced the false
> negative above.** Waiting on a screen that is asking you a question is an indefinite wait.

**Rule:** before reporting *any* CI product as unavailable, open its portal and complete whatever
it asks for. Only report "not installed" if the portal itself refuses — a licensing or entitlement
error, not an unanswered form.

This is the same failure mode as the rest of this skill: **a status surface reported a state that
was not true.** Here it cost another operator a working product.

---

## Licensing — CI is included, and it is ORG-level ✅ verified 2026-08-26

**The same operator was also told they were not licensed for CI. That was wrong too**, and the
reason is worth understanding because it will otherwise recur on every tenant.

### Finding 1 — CI ships inside the demo SKU

The **`Dynamics 365 and Power Platform multi-app demo`** SKU (20 seats on the reference tenant)
contains these service plans, read directly from *Licenses → the SKU → Manage apps & services*:

| Service plan | Assignment |
|---|---|
| **Dynamics 365 Customer Insights** | **Organization level** |
| **Dynamics 365 Customer Insights Accounts Add-on** | **Organization level** |
| **Dynamics 365 Customer Insights Engagement Insights** | **Organization level** |
| **Dataverse for Cust Insights BASE** | **Organization level** |
| **Dynamics 365 for Marketing** *(this is CI – Journeys)* | **Organization level** |
| Microsoft Dynamics 365 Customer Voice for Customer Insights App | Organization level |
| Dynamics 365 for Sales / Customer Service / Field Service | Per user |

### Finding 2 — why the licence check returns a false negative

Every CI plan above is labelled:

> *"This app is assigned at the organization level. It can't be assigned per user."*

**So inspecting a user's assigned licences will never show Customer Insights — on any tenant, even
a fully entitled one.** A per-user check is simply the wrong check. Sales, Customer Service and
Field Service *are* per-user, which makes the per-user approach look reliable right up until it
silently fails on CI.

### Finding 3 — "Your products" is not the licence list

Two pages in the same admin center disagree:

| Page | Showed |
|---|---|
| Billing → **Your products** | **1 item** — Microsoft Power Automate Free |
| Billing → **Licenses** | **17 items**, including the multi-app demo SKU |

**"Your products" reflects billing-account purchases**, and on a CDX tenant almost everything
arrives through a different path. Reading it as the entitlement list makes a fully licensed tenant
look empty. Use **Licenses**.

### ✅ The correct way to check CI entitlement

1. `https://admin.cloud.microsoft/#/licenses` — **not** `#/subscriptions`
2. Open **Dynamics 365 and Power Platform multi-app demo**
3. Select any licensed user → **Manage apps & services**
4. Look for the `Dynamics 365 Customer Insights` family

Or skip it entirely and **just open `home.ci.ai.dynamics.com`** — if it loads and binds to an
environment, you are licensed, whatever the admin center implies.

> **Never conclude "not licensed for CI" from a per-user licence check or from "Your products".**
> Both are structurally incapable of showing an org-level plan.

---

## Finding the install surface

Two obvious paths are **broken**. Do not waste time on them:

**1. `pac` cannot install it**

```
pac application install --application-name DynamicsMKT_AnchorSolution
```
```
Error: This application utilizes a Custom Install Experience through a Single Page
Application. Install for such apps is not supported by this API. Please utilize the
Power Platform Admin Center to initiate the install and configuration experience.
```

**2. PPAC's per-environment Applications page crashes**

`/manage/environments/{envid}/applications` →
*"It looks like something went wrong."*
`Error: Cannot read properties of undefined (reading 'toString')`

Reproducible client-side PPAC bug.

### The path that works

1. PPAC → **Products → Dynamics 365 apps** (tenant level, `/manage/applications`)
2. Find a card titled **"Dynamics 365 Marketing Application"**
   — there are ~20 identical cards; they are the different anchor solutions
   (`DynamicsMKT_AnchorSolution`, `MicrosoftDynamics_MktCompleteAnchorSolution`,
   `SMBMarketingDemoData`, `MicrosoftDynamics_LinkedInLeadGenAnchorSolution`, …).
   Any of them works — they all route to the same place.
3. **Manage** → dialog: *"We're taking you to the Dynamics 365 apps admin page"* → **Open**
4. Redirects to:

```
https://public-usa.mkt.dynamics.com/fre/cio/manage?tenantProductId=<guid>&geo=NA&lcid=1033
```

That is the **one-stop CI – Journeys + CI – Data installation management experience**. It
shows a per-environment grid with Install links for each product.

Bookmark that URL — going straight there next time skips the whole dance.

### Automation notes

| Surface | Selector reality |
|---|---|
| PPAC app tiles | Fluent `[role="group"].fui-Card`, **not** grid rows. Row-based locators find nothing. |
| CI portal Install controls | `<a>` elements, **not** `<button>`. Button locators time out. |
| Disabled state | Signalled by inline `style="... cursor: not-allowed"`, not `disabled` or `aria-disabled`. |

---

## Installing

### Read the confirmation dialog before clicking

> **"There is no way to cancel and reverse an installation mid-cycle."**

It also discloses that CI – Journeys **may transfer customer data outside the selected Azure
geography** via Azure CDN when using the **Asset library** and **Forms** features. You can
avoid that by hosting assets and forms elsewhere.

Check **"Trials available"** at the top of the portal. If it reads `0/0`, a trial licence is
not available and the install will consume **production** licensing.

Get explicit sign-off from whoever owns the tenant before proceeding. This is not a
reversible experiment.

### The two products install serially

**You cannot install both at once.** The moment CI – Journeys moves to `Installing`, the
CI – Data Install link greys out (`cursor: not-allowed`). Wait for Journeys to finish, then
start Data.

### Measured timing — CI Journeys took just over 3 hours

Observed on a Production environment (Aug 2026), start to finish:

| Elapsed | State |
|---|---|
| 0:00 | Clicked Install, portal shows `Installing` |
| ~1:10 | First marketing solutions appear (`MicrosoftDynamics_Marketing`) |
| ~2:00 | ~59 marketing solutions in, `Customer Insights - Journeys` app module present |
| 2:10 | Portal **still** shows `Installing` |
| **3:11** | Complete — portal flips to `Installed`, 96 marketing solutions |

Total visible solutions went **110 → 228** across the install; apps **22 → 24**.

On the tenant tested, the installed app's unique name was `msdyncrm_MarketingSMBApp`,
matching the `D365_SmbMarketing` licence template. Your tier may differ.

### CI – Data is *far* slower than Journeys — measured

Journeys took 3h11m. **CI – Data took roughly 13h54m wall clock**, and almost all of it was
invisible:

| Time (UTC) | State |
|---|---|
| ~00:40 | Install started (immediately after Journeys finished) |
| 00:40 → 13:38 | **~13 hours of nothing.** No new solutions. No portal change. |
| 13:38:21 | First CI-Data solution appears (`msdyn_CustomerInsights`) |
| 14:00:49 | `msdyn_CustomerInsightsDataTables` |
| 14:24:34 | `msdyn_CustomerInsightsAnchor` (CI Base) |
| **14:33:59** | Last write — `MicrosoftDynamics_StandardDataAnchorSolution`. Done. |

Solutions settled at **242**. The actual solution-writing phase was only ~55 minutes; the
other ~13 hours was queue / provisioning with **no observable signal anywhere**.

**Do not budget 2–3 hours for CI – Data and do not call it failed overnight.** Start it at
the end of a day and expect it the next morning. Budget **most of a working day per product**,
and remember they install serially.

### Confirming CI – Data is genuinely finished

Solution count alone is not enough — poll it until it stops moving:

```js
const base = Xrm.Utility.getGlobalContext().getClientUrl();
const count = async () => {
  const r = await fetch(base + "/api/data/v9.2/solutions?$select=uniquename,installedon&$filter=isvisible eq true&$top=5000",
    { headers: { 'OData-MaxVersion':'4.0','OData-Version':'4.0','Accept':'application/json' } });
  const v = (await r.json()).value || [];
  const latest = v.reduce((m,s) => new Date(s.installedon||0) > m.d ? { d:new Date(s.installedon), u:s.uniquename } : m, { d:new Date(0), u:null });
  return { n: v.length, latest: latest.u, on: latest.d.toISOString() };
};
// sample every 20s; stable count over ~80s = done
```

If the CI portal shows a first-run screen asking for **country/region** and consent, **fill it in —
that is the install step, not a progress indicator.** See the correction under "Checking what is
installed": treating this screen as transient is exactly what caused a false "not installed" report
for another operator. Once complete you land on the real home page bound to your environment name
with "Add data: Not started".

Only treat the portal as still settling if it shows a genuine spinner or an explicit
"provisioning" message with **nothing to answer**.

#### ✅ Verified end state (2026-08-26)

Both products confirmed landed and idle at zero content:

| Product | Check | Result |
|---|---|---|
| CI – Journeys | `appmodule` query in the org | ✅ `Customer Insights - Journeys` `[msdyncrm_MarketingSMBApp]` present, plus `Teams Virtual Events` `[msdynmkt_teamsvirtualeventscoms]`. Org app count 24. |
| CI – Journeys | content tables queryable | ✅ schema deployed — `msdyncrm_customerjourney`, `msdyncrm_marketingemail`, `msdyncrm_marketingform`, `msdyncrm_marketingpage`, `msdynmkt_journey`, `list` all resolve |
| CI – Journeys | content volume | **0 in every table** — bare install |
| CI – Data | portal home | ✅ `home.ci.ai.dynamics.com` bound to the environment, `instanceId` assigned |
| CI – Data | Getting-started checklist | **Add data: `Not started`**, Unify: *"Add data sources to proceed"* |
| CI – Data | content volume | 0 segments, 0 business measures, 0 enrichments |

**The CI – Data home screen's "Getting started" checklist is the authoritative progress surface.**
It is a three-state ladder — `Not started` → data sources added → `Unify data` unlocked. Reading it
takes one glance and it never lies, unlike the provisioning portal status.

> ⚠️ **But read it *after* the page hydrates.** Scraping `document.body.innerText` immediately after
> `browser_navigate` returns the **pre-hydration skeleton**, which renders every step as
> `Not started` regardless of true state. On a tenant with 408 unified customers, an early read
> reported `Add data: Not started`; a snapshot a moment later correctly showed
> `Add data: Completed` / `Unify data: Completed`.
>
> Use an accessibility snapshot or wait for a known-final string. **Do not** conclude from a fast
> `innerText` read that ingestion did not happen — that is the same false-negative trap as the
> region-form issue above, just caused by timing instead of by the wrong surface.

Two quick status probes worth keeping:

```js
// CI - Journeys content census, from any model-driven app page
for (const e of ['msdyncrm_customerjourney','msdyncrm_marketingemail','msdyncrm_marketingform',
                 'msdyncrm_marketingpage','msdynmkt_journey','list']) {
  try { console.log(e, (await Xrm.WebApi.retrieveMultipleRecords(e,'?$top=50')).entities.length); }
  catch (err) { console.log(e, 'ERR', err.message); }
}
```

⚠️ Observed on the CI – Data home page: the **Copilot (preview) pane reports "Failed to connect"**
on a freshly provisioned instance with no data sources. Cosmetic at this stage — there is nothing
for it to reason over. Re-check after ingestion rather than treating it as an install defect.

⚠️ Note `msdyncrm_segment` does **not** expose `msdyncrm_name`; querying that column throws
`Could not find a property named 'msdyncrm_name'`. Select `$top` only, or inspect the metadata
first. This is an easy false alarm when writing a status sweep.

### The portal status lags the org — check Dataverse for real progress

The provisioning portal keeps showing `Installing` well after the app module actually exists.
Do not sit watching it. Query the org instead:

```js
const base = Xrm.Utility.getGlobalContext().getClientUrl();
const g = async u => (await (await fetch(base+'/api/data/v9.2/'+u,
  {headers:{Accept:'application/json'}})).json()).value || [];

// marketing solutions landing, newest first
const sols = await g("solutions?$select=uniquename,version,installedon"
  + "&$filter=isvisible eq true&$top=400");
const mkt = sols.filter(s => /mkt|market|journey|customerinsight/i.test(s.uniquename));
console.log('marketing solutions:', mkt.length);
console.log('latest install:', mkt.map(s=>s.installedon).sort().pop());

// the definitive signal: does the app module exist yet?
const apps = await g('appmodules?$select=name&$top=100');
console.log(apps.map(a=>a.name).filter(n=>/insight|journey|market/i.test(n)));
```

Rising solution count with a recent `installedon` = healthy, keep waiting.
`Customer Insights - Journeys` appearing in `appmodules` = effectively landed.

Total solution count is a good coarse signal too — it went from **110 → 179** across this
install.

---

## CI – Data: storage and ingestion

### Output storage — no Azure subscription required

The CI – Data install dialog asks **"Save output data to"** and defaults to:

> **Customer Insights managed storage**

This is Microsoft-managed storage. **You do not need your own Azure Data Lake for CI – Data's
output.** Earlier guidance that CI – Data is hard-blocked without an Azure subscription is
wrong on this point — take the managed-storage default and the install proceeds.

(The alternative in that dropdown is your own Azure Data Lake Storage Gen2, which is what you
would use if a customer required data residency in their own tenant.)

### Ingestion sources — still worth planning

Output storage and *ingestion* are separate concerns. Data sources are configured after
install. Options that do **not** need an Azure subscription:

| Option | Notes |
|---|---|
| **Power Query** | Broad connector set — SharePoint, OneDrive, OData, SQL, web. Best fit for a demo tenant. |
| **Dataverse** | Read tables directly from the same environment. Most robust. |
| Azure Data Lake Gen2 | Needs an Azure subscription. Skip unless you have one. |

Confirm what the ingestion wizard offers in your build rather than assuming.

---

## ✅ Ingesting local CSVs via Power Query — verified walkthrough (2026-08-26)

The fastest route from generated CSVs to a live data source. **No Azure subscription, no Dataverse
tables, no storage account.** Whole thing takes ~15 minutes for five files.

Verified connector list on the "Add a data source" page: Fabric OneLake (Preview), Azure Data Lake
CDM tables, Azure Data Lake Delta tables, Azure Synapse Analytics (Preview), Microsoft Dataverse,
**Microsoft Power Query**, Data library.

### Put all tables in ONE data source

A single Power Query data source holds many queries, and **unification operates on tables, not on
data sources**. Five files → one source `CaldovaD2C` with five queries. Five separate sources would
mean five dataflows and five refresh schedules for no benefit.

Data source name rules: must begin with a letter, alphanumeric only, **no spaces or special
characters**, and **cannot be renamed after saving**.

### The flow

1. **Add data sources** → **Microsoft Power Query** → name it → **Next**
2. **Text/CSV** connector → Connection settings → switch Builder mode from `Link to file` to
   **`Upload file`**
3. **Browse…** → pick the CSV → wait for `Upload successful (NN KB)`
4. First file only: **Connection credentials** → Authentication kind `Organizational account` →
   **Sign in**. Subsequent files reuse the connection with no prompt.
5. **Next** → preview → **Transform data** (first query) or **Create** (subsequent queries)
6. In the editor: **Use first row as headers**, then fix data types (below)
7. Repeat step 1's *Get data* button inside the editor for each remaining file
8. **Save**

> ⚠️ **Upload file puts the CSV in the *tenant admin's* OneDrive for Business**, at
> `/personal/<admin>/Documents/Apps/Microsoft Power Query/Uploaded Files/`. Two consequences: the
> tenant needs OneDrive provisioned for that account (a CDX demo tenant does), and the demo data
> now lives in a user's mailbox-adjacent storage. Fine for fictional data; do not do this with
> anything real.

### ⚠️ Two things the wizard gets wrong, every time

**1. Headers are not auto-promoted.** A CSV imports as `Column1 … ColumnN` with the real header row
sitting in row 1 as data. You must click **Use first row as headers** on *every* query. Easy to
miss, and it silently produces a table whose columns are all named `ColumnN` — which then poisons
mapping and unification downstream.

**2. `Detect data type` does nothing.** Every column arrives as `Text`. Selecting all columns and
running Transform → **Detect data type** returns:

> *"Detect data types — We didn't detect any better data types than the ones you are already
> using."*

…and changes nothing. It will not infer dates or numbers from a promoted-header CSV. **You must set
types explicitly.**

This matters because CI – Data needs real types downstream: **Activities require a genuine
date/datetime field** for the customer timeline, and **measures require numeric columns**. Leaving
them as text blocks both.

**The upside, and keep it:** all-text default is *correct* for `PostalCode`, `Zip` and `Phone` —
it preserves leading zeros and formatting variants, which is exactly what the identity-resolution
demo depends on. Only fix dates and amounts; deliberately leave the rest as Text.

### Setting types explicitly

The **"Advanced Query Editing"** button is a *step-level* formula editor, not a full-query M editor
— it shows only the current step's expression. Use the formula bar instead:

**Insert step** → type into the **Formula bar** (no leading `=`) → **Enter**.

```m
Table.TransformColumnTypes(#"Promoted headers", {{"OrderDate", type date}, {"Quantity", Int64.Type}, {"UnitPrice", type number}, {"TotalAmount", type number}})
```

Verified type maps for the `gen-ci-data.py` output:

| Query | Transform |
|---|---|
| `CaldovaLoyaltyMembers` | `{{"BirthDate", type date}, {"EnrollmentDate", type date}, {"PointsBalance", Int64.Type}}` |
| `CaldovaEcommerceCustomers` | `{{"AccountCreatedOn", type date}}` |
| `CaldovaRebateRedemptions` | `{{"RedemptionDate", type date}, {"RebateAmount", type number}}` |
| `CaldovaTransactions` | `{{"OrderDate", type date}, {"Quantity", Int64.Type}, {"UnitPrice", type number}, {"DiscountPct", type number}, {"TotalAmount", type number}}` |
| `CaldovaWebActivity` | `{{"EventDate", type date}}` |

Queries are auto-named from the filename, so `CaldovaLoyaltyMembers.csv` → query
`CaldovaLoyaltyMembers`. Convenient — name the CSVs the way you want the tables named.

### After Save

`Validating queries` → the page returns to **Data sources** with the source listed as
`Power Platform | 5 tables | Refreshing`. First refresh of ~4,000 rows across five tables took
**more than 4 minutes**; let it run rather than re-triggering.

### Automation note

If driving this with Playwright, the **file chooser is intercepted by the MCP layer** — an in-page
`waitForEvent('filechooser')` will never fire. Click `Browse…`, let the tool surface the file
chooser, then satisfy it with the dedicated file-upload call.

> 🧹 **`gen-ci-data.py` writes its output next to itself** (`scripts/ci-data/`). If you run it from
> inside the skill folder it leaves ~430 KB of CSVs there, which then get packaged into any zip you
> build. Run it from a working directory, or delete `scripts/ci-data/` before packaging.

---

## Generating CI – Data demo data

Use `scripts/gen-ci-data.py`. It produces five CSVs modelling a CPG manufacturer with a
direct-to-consumer arm:

| File | Rows (default) | Role |
|---|---|---|
| `CaldovaLoyaltyMembers.csv` | 347 | Profile source — loyalty program |
| `CaldovaEcommerceCustomers.csv` | 323 | Profile source — D2C shop |
| `CaldovaRebateRedemptions.csv` | 711 | Activity + weak identity signal |
| `CaldovaTransactions.csv` | 1,467 | Activity — feeds measures |
| `CaldovaWebActivity.csv` | 1,176 | Activity — engagement |

### Why three profile sources

**A CDP demo where every record matches cleanly proves nothing.** The generator deliberately
builds in identity-resolution difficulty:

| Challenge | Default volume |
|---|---|
| People in two or more sources | 262 loyalty+ecom, 186 in all three |
| Source-exclusive people (must **not** merge) | 85 loyalty-only, 61 ecom-only |
| Phone format drift | `(469) 555-1187` / `469-555-1187` / `4695551187` |
| Nickname variants | 21 (Michael→Mike, Nicole→Nikki, Victor→Vic) |
| Casing / whitespace noise | uppercase and space-padded ecommerce names |
| Missing email | 58 rebate rows — forces phone-only matching |

**420 distinct real people** → that is the ground-truth unified profile count. Having a known
correct answer means you can prove on stage that unification worked.

The generator is deterministic (fixed seed). Edit `N_PEOPLE` and the "Coverage plan" block to
change scale and overlap. Brand names, products and retailers are pulled from the same CPG
portfolio used in `04-demo-data-reskin.md`, so the CDP data lines up with the Sales and
Service data instead of being a disconnected third dataset.

All generated emails use `@example.com`.

---

## Unification — executed and verified (2026-08-26)

### ⚠️ The wizard is no longer Map / Match / Merge

Current flow: **Customer data → Deduplication rules → Matching rules → Unified data view →
Review → Create customer profiles.** Older guidance describing "Map, Match, Merge" tabs is stale.

### What the wizard gets right on its own

**Intelligent mapping** assigned semantic types correctly with no intervention —
`EmailAddress → Identity.Service.Email`, `MobilePhone → Identity.Service.Phone`,
`FullName → Person.FullName`, `Zip → Location.PostalCode`, `City → Location.City`. 19 of 25
columns mapped automatically. Review it, don't redo it.

The **Unified data view** also auto-combined the overlapping columns sensibly (Email, Phone, City,
StateProvince, PostalCode each merged from both sources) leaving 13 single-source columns.

### Four things to set by hand

1. **Primary key per table.** Required, and the dropdown **excludes date and numeric columns** —
   so a table keyed on something typed as a date will appear to have no valid key.
2. **Table order.** The first table is the primary profile source and should be the most complete
   and trusted. Loyalty (14 columns, incl. tier, points, birth date) outranks Ecommerce here, so
   move it to position 1 — the wizard orders alphabetically by default, which is meaningless.
3. **"Include all records"** — retains singletons (people present in only one source). Defaults to
   **checked**; leave it. Unchecking it silently deletes every single-source customer.
4. **Matching rules** — at least one, between table 1 and table 2.

### ⚠️ There is no normalisation option in the matching-rule dialog

The Add rule dialog offers only: table, field, table, field, **Precision** (`Basic` → Exact / High
/ Medium / Low) and a name. **There is no "Normalize" control** (no ignore-symbols, no
numerals-only).

This directly contradicts the older plan below, which assumed you could normalise phone inside a
match rule. **You cannot.** If your sources format phone numbers differently — as these
deliberately do, `(469) 555-1187` vs `469-555-1187` vs `4695551187` — you must add a normalised
digits-only column **upstream in Power Query** and match on that. Budget a full re-refresh
(~19 min) if you discover this late.

Verified working rule: `CaldovaLoyaltyMembers.Email` = `CaldovaEcommerceCustomers.EmailAddress`,
Precision `Exact`.

### ⚠️ Recompute the ground-truth target for YOUR source roles

`gen-ci-data.py` prints `expected unified profiles 420`. **That number assumes all three sources
are profile sources** — which contradicts this file's own guidance to treat rebates as an activity.

With the recommended two-profile-source configuration the correct target is **408**:

```
loyalty 347  +  ecommerce 323  −  in both 262  =  408
420 distinct people − 408 = 12 people who exist ONLY in rebate redemptions
```

Those 12 have no profile, and that is **correct** — they are anonymous rebate redeemers with no
customer record. It is also a good demo talking point rather than a defect. Do not chase 420.

**Fix the generator's summary line, or read it as "if you unify all three".**

### ✅ Verified result — an exact match to ground truth

Running with a single `Email` exact rule, two profile sources, singletons retained:

| Metric | Expected | Actual |
|---|---|---|
| Unified customer profiles | 408 | **408** ✅ |
| Matched records (loyalty ∩ ecommerce) | 262 | **262** ✅ |
| Duplicates found | 0 | **0** ✅ |

Both numbers landed exactly, which means **no over-merging and no under-merging**. A single exact
email rule was sufficient for profile unification — the phone-format divergence baked into the
generator is a problem for **activity linking** (attaching rebate rows), not for matching these two
profile sources, because every loyalty and ecommerce row carries a well-formed email.

That distinction is worth keeping: *normalise where the join is weak, not everywhere.* Adding a
fuzzy name or phone rule here would have risked degrading a result that is already exact.

Timing: `Create customer profiles` → matching complete in **~4 minutes**; the Unified data view
step continues a little longer.

### Automation note

---

## Planned unification configuration

> Superseded in part by the verified section above — the phone-normalisation step in "Match order"
> is **not** achievable inside a matching rule.

### Source roles

| Source | Role | Key |
|---|---|---|
| Loyalty | Profile | `LoyaltyId` |
| Ecommerce | Profile | `CustomerId` |
| Rebate redemptions | **Activity** (+ match signal) | `RedemptionId` |
| Transactions | **Activity** | `TransactionId` |
| Web activity | **Activity** | `ActivityId` |

Rebates and transactions are events, not customer records. Configure them as activity data
linked to the unified profile — do not add them as a third profile source, or you will
inflate the profile count.

### Match order

1. Loyalty → Ecommerce on `Email` = `EmailAddress` (exact, case-insensitive)
2. Loyalty → Rebate on `Email` = `Email` (exact)
3. Ecommerce → Rebate on **normalised** `Phone` — required for the 58 email-less rows
4. Optional fuzzy pass on `Name + PostalCode` to catch nickname variants

Phone normalisation is not optional here; the three sources use three different formats by
design.

### Measures

Lifetime value (`SUM(TotalAmount)`), order count, average order value, days since last order,
rebate redemption count, category affinity (top category by spend).

### Segments

High-value loyalists · Lapsing customers · Energy drinkers · Household & personal care buyers ·
Rebate-driven low-D2C · Opted-in never purchased · Club-pack shoppers.

Full definitions in the generated `ci-data/README.md`.

Fluent checkboxes in this wizard cannot be clicked directly — the `<label>` intercepts pointer
events. Click `label[for="<checkbox id>"]` instead.

---

## Validating unification

After the first unification run, check the profile count against the generator's reported
"distinct real people" figure.

- **Materially fewer** → over-merging. Usually a too-loose fuzzy name rule.
- **Materially more** → under-merging. Usually phone normalisation missing, so the
  email-less rebate rows failed to attach.

This is the main reason to use a generator with a known ground truth rather than random data.

---

# CI – Journeys: authoring emails via API ⚠️ partially blocked (2026-08-26)

> **Verification status.** Everything below was executed against a live tenant. The conclusion is
> a **negative result with a precise boundary**: content can be authored via API, but the final
> `Draft → Ready to send` transition cannot be completed by API alone.

## The single most useful technique: `msdynmkt_ValidateEmail`

Publishing failures surface as useless text. All three of these describe the *same* email:

| Surface | What you get |
|---|---|
| UI notification | *"Please check validation errors"* |
| `Xrm.WebApi.updateRecord(... statuscode:2)` | *"Validation returned errors. It is not possible to update state to ready to send."* |
| **`msdynmkt_ValidateEmail`** | **a structured, per-field report naming every failure** |

**Always call the action.** It returns `ValidateResult` as a JSON *string* holding
`placeholders.result` and `message.result`, each keyed by field with `valid` and a `states[]` array
carrying `code`, `severity` and a message template.

```js
const e = await Xrm.WebApi.retrieveRecord('msdynmkt_email', id);
const req = {
  message: {
    to:          { content: e.msdynmkt_to },
    replyToEmail:{ content: e.msdynmkt_fromemail },
    textContent: { content: 't' },
    htmlContent: { content: e.msdynmkt_emailbody },
    fromEmail:   { content: e.msdynmkt_fromemail },
    fromName:    { content: e.msdynmkt_fromname },
    subject:     { content: e.msdynmkt_subject }
  },
  messageDesignation: 534120000,
  complianceSettingsId: e._msdynmkt_compliancesettings4_value,
  purposeId: e._msdynmkt_purpose_value,
  topicId: null,
  placeholders: JSON.parse(e.msdynmkt_placeholders || '{}'),
  conditionalContentData: null,
  gatherStatistics: false,
  context: { trigger: 1, owningBusinessUnitIds: [] },
  messageId: id,
  emailContentType: 534120000
};
const r = await fetch('/api/data/v9.0/msdynmkt_ValidateEmail', {
  method:'POST', headers:{'Content-Type':'application/json','Accept':'application/json'},
  body: JSON.stringify({ ValidateRequest: JSON.stringify(req) })
});
const res = JSON.parse((await r.json()).ValidateResult);
```

**How the payload shape was obtained** — a reusable trick when an action is undocumented: click the
real button in the UI, then read the outgoing request body from the network log. Guessing the shape
returns a bare `400 Bad Request` with no hint.

---

## Three fields the designer owns, and what each is for

| Field | Role |
|---|---|
| **`msdynmkt_designerhtml`** | **source of truth** — designer block markup |
| `msdynmkt_emailbody` | **compiled output** — email-client-safe HTML, regenerated from designerhtml |
| `msdynmkt_placeholders` | **derived** — auto-populated on save by scanning the body for `{{tokens}}` |

**Writing `msdynmkt_emailbody` alone accomplishes nothing.** Nine emails were seeded with 1.6–2.1 KB
of hand-written HTML in `emailbody`, with `designerhtml` empty. The designer rendered **only the
compliance footer** — the authored content was invisible because the designer builds its canvas from
`designerhtml`, not from `emailbody`.

Opening one of those emails and pressing **Save** regenerated everything:

| Field | Before | After Save |
|---|---|---|
| `msdynmkt_emailbody` | 1,993 | **10,943** (designer-compiled) |
| `msdynmkt_designerhtml` | 0 | **8,591** |
| `msdynmkt_placeholders` | `null` | auto-derived JSON |

**The authored content was destroyed by that save.** Back up any hand-written body before letting
the designer touch a record.

### The block format

Content lives in `<th class="inner ui-sortable">` cells inside `data-section` divs:

```html
<body dir="ltr">
  <div data-layout="true" data-layout-version="v2" style="max-width: 600px; margin: auto;">
    <div data-section="true" class="columns-equal-class wrap-section emptySection">
      ... <th data-container="true" class="columnContainer" data-container-width="100" id="container...">
            ... <th class="inner ui-sortable" style="...">
                  <!-- elements go here -->
                  <div data-editorblocktype="Text" style="margin: 10px;">
                    <h1>Heading</h1><p>Body copy.</p>
                  </div>
```

`data-editorblocktype` values include `Text`, `Image`, `Button`, `Divider`. The designer **does**
render blocks injected this way — verified: injected Text blocks appeared correctly in the canvas
and in the validator payload.

**Working pattern:** save one email through the designer once, read its `designerhtml` as a
skeleton, then string-inject `data-editorblocktype` blocks into the empty cell for every other
email. Nine emails were authored this way in a single API pass.

---

## `statuscode`, not a publish action

| Entity | Draft | Live/Ready |
|---|---|---|
| `msdynmkt_email` | `statuscode: 1` | **`statuscode: 2`** = *Ready to send* |
| `msdynmkt_journey` | `statuscode: 1` | **`statuscode: 2`** = *Live* (also 4 Publishing, 8 Completed, 10 Paused) |

There is no `PublishEmail` action; you flip `statuscode`. Journeys **do** have
`msdynmkt_PublishJourney` / `msdynmkt_PublishJourneyV2`.

---

## Publish prerequisites, in the order they surface

**1. `msdynmkt_placeholders` must not be null** → *"Placeholders cannot be null or empty."*
`[]` and `{"k":"v"}` both give a bare `400`. `{}` is accepted. The real value is the
designer-derived JSON.

**2. `msdynmkt_purpose` is mandatory** → `MandatoryPurposeMissing`, *"Purpose field is mandatory."*
Not enforced by the form, and easy to miss. Seeded purposes:

| Purpose | Id |
|---|---|
| Transactional | `<guid>` |
| **Commercial** | `<guid>` |
| Tracking | `<guid>` |

```js
await Xrm.WebApi.updateRecord('msdynmkt_email', id, {
  'msdynmkt_purpose@odata.bind': '/msdynmkt_purposes(<guid>)'
});
```

**3. A compliance profile must be attached** — bind `msdynmkt_compliancesettings4`. Note the entity
is `msdynmkt_compliancesettings4` (set: `msdynmkt_compliancesettings4s`). `msdynmkt_complianceprofile`
and `msdynmkt_senderprofile` **do not exist**; `msdynmkt_compliancesettings` is a *different, older*
entity that also exists. Guessing these names wastes a round trip.

**4. ⛔ Compliance placeholders must be BOUND — designer-only.**
`{{CompanyAddress}}` and `{{PreferenceCenter}}` fail with
`NoBinding` — *"The data source has not been set correctly."*

**Filling the compliance profile does not clear this.** Setting `msdynmkt_legaladdress`,
`msdynmkt_subscriptioncenter` and `msdynmkt_consentlink` on the profile left the error unchanged.
The binding lives **in the placeholder JSON itself**, and the API-written entry lacks it:

```json
{"CompanyAddress":{"title":"CompanyAddress","defaultValue":null,
 "type":["null","string"],"category":"general","isInternal":false,"touched":false}}
```

Note `"touched": false` — the designer sets the binding when the token is configured in the UI.

**5. ⛔ `msdynmkt_to` fails standalone validation.**
`{{contact.emailaddress1}}` returns `UnsupportedProperty` — *"Dynamic content contains an invalid
property."* This is the designer's own default value. It appears to resolve from **journey audience
context**, so a standalone validate with `context.trigger: 1` rejects what the journey would supply.

### Confirmed dead end for API-only publishing

Stripping the entire compliance footer and switching to **Transactional** purpose (which does not
require an unsubscribe link) still failed — blocker **5** is independent of the footer.

**Conclusion: emails can be fully authored via API, but each must be opened in the designer once to
bind compliance placeholders before it will go Ready to send.** Budget roughly a minute per email
of UI work, and script everything else.

---

## Infrastructure that was already correct

Worth knowing so it is not misdiagnosed as the problem:

| Component | State |
|---|---|
| Sending domain | ✅ `unqc5e7931ce59ff1118add000d3a5cb.s05.dyn365mktg.com` — **auto-provisioned and validated**, no DNS work |
| Brand profile + sender | ✅ `Default brand profile` / `Default brand sender` |
| Compliance profiles | ⚠️ two exist (`default`, `pes`) but both ship as **stubs** — `legaladdress: "Address"`, `subscriptioncenter: null` |
| Preference center | ⚠️ `default Preference Center` exists but is **not linked** to either profile |

**A CDX tenant gives you a verified sending domain for free** — a genuinely useful demo asset, since
domain validation is normally the slowest part of standing up marketing email.

---

## Segment DSL — which queries actually go live

On the reference tenant, **2 of 10 segments** reached *Ready to use*. The split is exact:

| Status | Query |
|---|---|
| ✅ Ready to use | `PROFILE(contact).FILTER(jobtitle == 'Consumer' && donotbulkemail == false)` |
| ✅ Ready to use | `PROFILE(contact).FILTER(jobtitle == 'Demo Consumer')` |
| ❌ Draft | `...FILTER(contains(emailaddress1,'.example'))` |
| ❌ Draft | `...FILTER(contains(jobtitle,'Buyer') \|\| contains(jobtitle,'Category'))` |

**Only `==` and `&&` survive. `contains()` and `||` leave the segment stuck in Draft.** Design
segments around exact-match fields — which means **seeding contacts with clean, exact-match values
in `jobtitle` is what makes segmentation demoable**. Plan that during data generation, not after.

`msdynmkt_PublishSegmentDefinition` (parameter `SegmentId`) is the publish action.

---

# CI – Data: segments and activities ⚠️ partially blocked (2026-08-26)

## ⛔ Activities did NOT persist — verify, do not trust the wizard

An earlier pass reported **5 of 9 activity tables configured**. Returning later, the Activities page
showed the empty state: *"Add activity data to your customer profiles."* **Nothing had saved.**

This is the skill's own core lesson biting from the inside: **a wizard step that appeared to
complete had not.** Always re-open the Activities page in a fresh session and confirm rows exist
before building anything on top of them.

## Segment builder — attribute picking works, Save gate is unreliable

**What worked, reliably:**

1. `Insights → Segments → New segment` opens `/app/segments/customer/builder`.
2. **Add condition** creates a rule row.
3. The attribute combobox (`placeholder="Enter an attribute name or add from side panel"`) filters
   as you type and returns options labelled **`Table: Customer | Data source: ConflationResolve…`**
   (the unified profile) alongside per-source copies.
   **Always pick the `Table: Customer` option** — that is the unified entity; the source-table copies
   segment only that source.
4. Choosing an attribute auto-populates operator (`equal to`) and **guesses a value** from real data
   — it offered `Bronze` for `LoyaltyTier`, proving the profile store is populated.
5. `Edit details` sets Name, **Output table name** *(immutable once saved — get it right)*,
   Description and Tags. Values applied cleanly and the header updated.

**Where it stalls:** the **Save** button (`button[data-testid="save"]`) stays `disabled` even with a
complete rule and a name. Retyping the value through a real `fill()` did not release it.

This matches the documented CI wizard failure mode — **panels do not re-render and hold stale
internal state**, the same reason left-nav table switching in the Activities wizard is unreliable.

**Recommendation: build segments by hand in the UI.** The React state machine resists synthetic
events, and a segment takes under a minute manually. Automate CSV ingestion and unification —
those are fully scriptable and verified — and hand-build the handful of segments on top.

### Attribute-naming payoff

`LoyaltyTier`, `PointsBalance`, `PreferredRetailer`, `EmailOptIn` and `EnrollmentDate` all surfaced
cleanly as pickable unified attributes. **Segment quality is decided during data generation**, not in
the builder: seed clean, exact-match, low-cardinality values (a 4-value tier column) and the segments
almost write themselves. This is the same conclusion the CI – Journeys segment DSL forced.

---

## 🚨 API-created emails cannot go live — the personalization-binding trap

**Verified 2026-08-26. This is the single hardest blocker in CI – Journeys, and it only bites when
you author emails programmatically.**

Emails created through the Web API sit at `statecode 0 / statuscode 1` (Draft) and **the
`Ready to send` button silently does nothing**. Any journey referencing them fails Publish with:

```
Errors in Journey: 3
A message marked draft cannot be sent. Go to the email and choose ready to send to use it.
```

### Diagnose it with `Check content`

The email's own **Check content** validator is the only surface that names the cause. It reports:

| Severity | Message |
|---|---|
| ❌ Error | `To: Dynamic content contains an invalid property: {{contact.emailaddress1}}.` |
| ❌ Error | `Placeholder "CompanyAddress": The data source has not been set correctly.` |
| ❌ Error | `Placeholder "PreferenceCenter": The data source has not been set correctly.` |
| ⚠️ Warning | Text part / Email body should contain the "Company address" placeholder |
| ⚠️ Warning | Text part / Email body should contain the "Preference center" placeholder |

> The notification bell shows a count but the flyout will not open under scripted `.click()`.
> Take a **screenshot** — the banner list renders fine visually even when the DOM read comes back
> empty.

### Root cause: declared tokens, unbound data sources

Read `msdynmkt_placeholders` on the email:

```json
"CompanyAddress":  { "title":"CompanyAddress",  "defaultValue":null, "touched":false, ... },
"PreferenceCenter":{ "title":"PreferenceCenter","defaultValue":null, "touched":false, ... }
```

The tokens are **declared but never bound**. `defaultValue:null` + `touched:false` is the signature.
`msdynmkt_to` has the same problem — `{{contact.emailaddress1}}` is a string with no registered
source behind it.

**The designer's personalization dialog is what writes those bindings.** An email that never passed
through it cannot validate, no matter how correct the HTML looks.

### What is NOT the cause — do not chase these

| Suspect | Reality |
|---|---|
| Missing compliance profile | Profile `default` was fully configured — real legal address **and** preference-centre URL |
| Email not bound to the profile | Every email already had `_msdynmkt_compliancesettings4_value` set correctly |
| Missing company address value | `msdynmkt_compliancesettingscompanyaddress` held the real address |

> ⚠️ **Do not "fix" it by assigning a brand profile.** It does **not** clear the errors, and it
> **silently overwrites your sender** — `Caldova Rewards <rewards@…>` became
> `Default brand sender <admin@…>`. Reverting takes two calls: `DELETE`
> `msdynmkt_emails(<id>)/msdynmkt_brandprofileid/$ref`, then PATCH `msdynmkt_fromname` /
> `msdynmkt_fromemail` back.

### The fix

Re-author the footer placeholders and the To binding **through the email designer UI** on each email
the journey uses, so the bindings register. Budget 15–30 minutes.

**Better: plan for this before you generate content.** If the demo needs a live send, build the
handful of journey emails by hand in the designer from the start and reserve the API for bulk
"library filler" emails that only ever need to look right in a list view.

## Journey go-live checklist

Two things to change on a journey built for a live send test, both easy to miss:

1. **Audience** — a journey built against a production-sized segment will send to all of it.
   Swap to the plus-addressed test segment. The lookup shows member counts
   (`DEMO - Live send test list, Contact (40 members)`), so verify the number before publishing.
2. **Start date** — journeys default to a future start. `Reset start time` snaps it to now, but the
   validator then says *"Choose a time in the future"* because the journey clock is **UTC**. The
   hour dropdown offers only whole hours; the field accepts a typed value like `10:50 PM`.

**Verify the recipients in Dataverse, not in the segment UI**, before any live send:

```js
const c = await raw("contacts?$select=fullname,emailaddress1&$filter=jobtitle eq 'Demo Consumer'");
const safe = /^{{USER_ALIAS}}\+cald\d+@microsoft\.com$/i;
c.filter(x => !safe.test(x.emailaddress1));   // must be empty
```

---

## 🚨 "The live send didn't work" — check Junk before you debug anything

**Verified 2026-08-27. This cost a full debugging cycle and the platform was never at fault.**

A published journey reported **Sent 40 · Delivered 40 · Blocked 0 · Delivery failed 0**, and the
target Outlook folder was **empty**. Both a journey send and a manual **Test send** appeared to
vanish.

**They were in the Junk Email folder**, subject-tagged `[EXTERNAL]`, carrying Outlook's
first-contact banner:

```
You don't often get email from rewards@unq….s05.dyn365mktg.com. Learn why this is important
```

### Why the demo rig's folder stayed empty

**Junk filtering happens BEFORE inbox rules.** The plus-addressing rig (see the demo send rig
section) files mail into a folder with an inbox rule — but a message routed to Junk never reaches
rule processing. The folder reads `0` and everything looks broken.

> **Check Junk Email and the Defender quarantine before concluding a send failed.** A brand-new
> `*.dyn365mktg.com` sending subdomain has no reputation, and bulk marketing mail into a
> Microsoft corporate mailbox is exactly what the filters are tuned to catch.

Symptom check that saves the cycle: if the journey's own analytics show **Delivered** with
**0 blocked / 0 failed**, the mail left Dynamics successfully. The problem is on the receiving side,
not in CI – Journeys.

**Fix for a repeatable demo:** add the sending address to **Safe Senders** in Outlook before the
demo. Only some of the run may reach Junk — in this build the 40 first-touch welcome emails never
appeared at all (almost certainly Defender-quarantined as first contact) while the 39 follow-ups
landed in Junk once the domain had a little reputation.

### ⚠️ Engagement metrics against Microsoft recipients are not real

The same run reported:

| Metric | Value |
|---|---|
| Open rate | **97.5%** |
| Unique opens | 39 of 40 |
| Total opens | 44 |
| Opens by email client | **44 "Unknown"** |

Nobody opened anything — the mail was sitting unread in Junk. **Microsoft Defender's safe-links and
attachment scanning fetched the tracking pixel**, and Dynamics counted each fetch as an open.

Two consequences worth knowing before you demo analytics:

1. **A ~100% open rate with an "Unknown" email client is the signature of security scanning**, not
   engagement. Do not present it as a real result.
2. **It changes journey behaviour.** Our branch condition was *"Previous message gets an
   interaction → Email Opened"*, so **39 of 40 contacts were routed down the engaged path by
   Defender**, not by people. If you are demonstrating a branching journey, either expect this or
   branch on **Email Link Clicked**, which scanning is far less likely to trigger.

---

## 🚨 A measure or segment must be ACTIVATED before it can refresh

**Verified 2026-08-27.** A newly saved measure or segment lands as:

```
State: Inactive · Status: Not started · Refreshed: Never
```

and **the `Refresh` command is greyed out** while it is Inactive. Clicking Refresh does nothing and
reports nothing, so it looks like the refresh simply failed. An overnight wait does not help either
— the "System refresh" schedule does not pick up an inactive object.

**Select the row → `Activate` → then `Refresh`.** Both live on the command bar once a row is
selected:

| Step | Result |
|---|---|
| Save | `Inactive · Not started · Never` — Refresh disabled |
| Activate | `Active · Not started` — Refresh now enabled |
| Refresh | `Refreshing` → `Successful` with a row count |

Verified end state: measure `Total customer spend` **Successful**, segment
`Platinum loyalty members` **Successful, 16 members**.

> ⚠️ Do not confuse the **row-level** `Refresh` button with the command-bar one. The row's inline
> control sits next to `Edit` and a mis-aimed click opens the editor instead — which looks like the
> refresh silently failing.

## ✅ Correction — segments DO save under automation

An earlier note in this file said the segment builder's **Save** stays disabled under automation and
recommended building segments by hand. **That does not reproduce.** The segment above was built and
saved entirely through automation.

The difference is *how the value is set*: **pick the value from the dropdown** rather than typing it
into the field. Choosing `LoyaltyTier` → operator `equal to` → then selecting `Platinum` from the
offered list left Save fully enabled.

Two things that help:

- Always choose the attribute option labelled **`Table: Customer | Data source: ConflationResolve…`**
  — that is the unified profile. The per-source copies segment only that one source.
- The value control is **single-select**, so one tier per rule. Use `Add condition` for more.

That the dropdown offered all four real tiers (Bronze / Gold / Platinum / Silver) is also a useful
signal in its own right: it proves the unified profile store is genuinely populated.

## 🔴 Prediction models only accept *semantic* activity types

**The trap.** Out-of-the-box prediction models do not let you pick an arbitrary activity. They ask
for a **semantic activity type**, and only activities created with that exact type are offered.

| Model | Activity type it demands |
|---|---|
| Transaction churn | `SalesOrder` or `SalesOrderLine` |
| Subscription churn | `Subscription` |
| Customer lifetime value | `SalesOrder` / `SalesOrderLine` |
| Product recommendations | `SalesOrderLine` |

Caldova's `CaldovaTransactions` activity was originally created as type **`Purchase`** — a perfectly
reasonable label, and it renders on the timeline exactly the same. But in the churn wizard the table
picker returned:

> There are no available tables for the selected activity type.

Nothing is wrong with the data. The activity is simply *invisible* to the model because of its type.

**So: choose the activity type with the prediction models in mind, not the timeline caption.**
`SalesOrder` and `SalesOrderLine` are deliberately the first two entries in the 37-item type list.

### Changing the type after the fact

The Activities **grid** offers only `Rename` and `Delete` when a row is selected — there is no Edit,
which makes the type look immutable. It is not. Reopen the wizard with the **`Configure activities`**
command (visible only when *no* row is selected) and the `Activity type` combobox on step 1 is fully
editable.

Changing the type is non-destructive: field mappings (Timestamp / Event activity / Web address /
Additional detail), the primary key and the relationship path all survive. Walk `Next` through the
three steps and `Save and close`.

**Then press `Run` on the Activities page.** The grid shows the new type immediately, but the change
is not live for the models until the activity job re-runs (~10–15 min). While it runs the page warns
*"Some actions are not available when activity processes are running."*

### Wizard automation notes (churn model)

- Correct URL is `/app/predictions` — **not** `/app/intelligence/predictions`.
- The model tile's `Use this model` opens a panel that is itself covered by `.ms-Overlay`; click the
  tile button via the **accessibility ref**, then work inside the panel.
- Subscription-vs-Transaction is a Fluent `ChoiceGroup`: the `<input type=radio>` is intercepted by
  its own label, so click **`label[for="<radioId>"]`**.
- Prediction-window / churn-definition are `role="spinbutton"` and ignore synthetic events — use a
  real Playwright `fill()`.
- Six steps: Model name → Preferences → Purchase history (required) → Additional data (optional) →
  Data updates → Review and run.

## 🔴 The *semantic field map* — the step that unlocks measures templates AND models

This is the single most important CI-Data finding of the build, and it is easy to miss entirely.

When you create an activity from **Data > Activities**, step 2 (`Activity fields`) only asks for four
generic display fields:

| Slot | Caldova mapping |
|---|---|
| Timestamp | `OrderDate` |
| Event activity | `ProductName` |
| Web address | *(blank)* |
| Additional detail | `Category` |

Those four drive the **timeline card** and nothing else. Below them sits an easily-overlooked prompt:

> **Map field types for your activity's attributes?**  ( Yes / No )

Answering **Yes** reveals the *semantic* slots for the chosen activity type. For `SalesOrder`:

| Semantic slot | Caldova mapping |
|---|---|
| Sales order ID | `TransactionId` |
| Order date | `OrderDate` |
| **Sales amount** | **`TotalAmount`** |
| Store ID | *(left blank)* |
| Is return? | *(left blank)* |

**Nothing downstream can see a column you did not map here.** Two concrete symptoms we hit:

1. **Measure templates could not find the money.** `Average transaction value (ATV)` asks for
   *"Monetary value of transaction"*, and the picker offered only the `UnifiedActivity` projection —
   `ActivityTime`, `Title`, `EventDate`, `Duration`, `ImageURL`… and **no `TotalAmount`**. The
   template is not reading your source table; it reads the projected activity. No semantic map, no
   amount.
2. **The churn model's `Save and run` stayed disabled**, with the review page reporting
   `Customer transaction history: In progress`.

### The path that actually works

The prediction wizard will drive this for you, and it is the most reliable route:

```
Predictions → Customer churn → Transaction
  → Purchase history (required) → Add data
  → Activity type = SalesOrder
  → [Edit]  ("Navigating to set up your activity" → Save draft)
  → activity wizard opens with the SEMANTIC slots exposed
  → map Sales order ID / Order date / Sales amount
  → Next → Next → Save and close
  → back in the prediction: Add data → SalesOrder → tick the activity → Next
  → Step 2 of 2 now validates
```

Note the ordering trap: **before** the semantic map exists, the activity checkbox on
`Step 1 of 2` renders **disabled** and `Next` stays greyed. That looks like "no data", but it
actually means "this activity is not semantically complete yet". Click `Edit`, not `Cancel`.

> Saving the activity from inside the prediction wizard **kicks off an activity re-run**, and
> `Save and run` on the review page stays disabled until it finishes (~10–15 min). Use `Save draft`,
> wait for `Data > Activities` to read `Successful`, then reopen the model and run it.

## ✅ Better: `Run` in the measure builder both saves *and* activates

The earlier "Activate before Refresh" note still holds for measures created with **`Save and close`**
— those land `Inactive` and their `Refresh` is greyed out.

But the measure builder's **`Run`** button does all three things at once: saves the measure, sets it
`Active`, and queues the refresh. Prefer it. Five Caldova measures were created this way and every
one landed `Active` with no separate activation step.

Also note the builder is reachable directly at **`/app/measures/builder?instanceId=…`**, which skips
the flaky `New ▸ Build your own` menu entirely. (`/app/measures/new` is *not* valid — it 404s to
"Oops! The page you are looking for does not exist".)

### Measure builder automation notes

- The `New` menubar and its `Build your own` item frequently ignore scripted `.click()` — drive them
  with accessibility refs, or just navigate to the builder URL.
- Buttons carry icon glyphs inside their text, so **match with `includes()`, never `=== 'Add attribute'`**.
- The attribute picker's radio must be clicked on its **label wrapper via an accessibility ref**;
  clicking the `[role="radio"]` input directly leaves `Add` disabled.
- `Edit details` holds Name + Output table name + Description. `Description` is a **`<textarea>`**, so
  a React value-setter written against `HTMLInputElement.prototype` throws `Illegal invocation` —
  branch on `el.tagName`.

## CI compute is a single serialized queue

Activities, measures, segments and prediction models all share one job queue per instance. Symptoms
worth recognising rather than debugging:

- While an activity job runs the Activities page warns *"Some actions are not available when
  activity processes are running"*, and **measure refreshes will not start**.
- A measure can read `Status = Not started` while `Refreshed` shows a recent timestamp. That is the
  queue, not a failure.
- Queueing a prediction pushes every pending measure behind it, and models are explicitly documented
  as taking *"a few hours at least"*.

So sequence the build deliberately:

```
load data → unify → activities (+ semantic map) → RUN and wait for Successful
          → measures + segments  (they pick themselves up automatically)
          → predictions last
```

**Don't chase "Queued".** Confirm instead that `State = Active` — that is the configuration state you
control. `Status` is just the scheduler.

## Segments: the three tabs — All segments / Suggestions / Insights

`Insights ▸ Segments` has **three tabs**, and the two preview ones are easy to miss:

| Tab | What it does |
|---|---|
| **All segments** | the segment list you build by hand |
| **Suggestions (preview)** | AI proposes segments from activity data (RFM) |
| **Insights (preview)** | **Differentiators** and **Overlap** analyses across segments |

The `New` menu is the entry point to all of them:
`Build your own · Create from Profiles · Create from Measures · Suggested segments · Segment insights`.

### `Create from Measures` — the fastest way to build a segment

Opens a **"New quick segment"** panel: pick a measure → operator (`greater than` / `less than` /
`between`) → value. It renders a **percentile distribution chart** for the chosen measure, which is
genuinely useful for choosing a sane threshold instead of guessing.

> ⚠️ **The "Estimated segment size" is an estimate, and it can be well off.** Caldova's
> `Total customer spend > 200` previewed **79 customers (19.36%)** and resolved to **50** on refresh.
> Fine for a demo, but don't quote the preview number as fact.

### Segment suggestions — a 4-step RFM wizard

`Get suggestions` → choose **"See or anticipate customer behavior"** (the other option is
"Improve a measure/metric") → then **Customers → Activity → Preferences → Review and run**.

The Preferences step is textbook **RFM**: Recency, Frequency, Monetary (all three ticked by default),
plus attribute mapping for `Timestamp` and `Value`.

> 🔴 **The trap: `Run` stays disabled with no visible reason.** The Activity step has a *second*
> control below the type picker — **"Confirm the applicable activity source table"** — that only
> renders **after** you pick the activity type. Advance too quickly and you skip it. The review page
> then shows `Activity source table:` **blank**, and `Run` is greyed with no error message.
> Go back to the Activity step and tick the source table checkbox.

### Segment insights — Differentiators and Overlap

Two types:

- **Differentiators** — what distinguishes one segment from another (or from everyone else)
- **Overlap** — shared membership between segments

Flow: primary segment → comparison segment → attributes → name → Save.

Two hard requirements that produce confusing symptoms:

1. **You need at least two segments whose refresh has COMPLETED.** With one segment refreshed, the
   comparison step reads *"No item found. Could not find any segments."* — which reads like a bug but
   just means the other segment was still `Refreshing`. Wait for `Successful`.
2. **🔴 Date-typed measures are offered but rejected.** The attribute picker happily lists a
   datetime measure, then `Save` fails with:
   ```
   'Differentiator' insight has a table 'Customer_Measure' with an attribute
   'Lastpurchasedate' that isn't allowed.
   ```
   **Deselect any date measure** (Caldova's `Last purchase date`). Numeric measures and text/choice
   customer fields are fine.

> Note the error toast **persists on screen after a later successful save**, still showing the old
> Request ID. Confirm against the insights **grid**, not the toast.

### Automation notes

- The **`New` menubar item ignores scripted `.click()`** — drive it with an accessibility ref. The
  menu then closes if a follow-up script throws, so chaining several segment builds in one
  `evaluate` is unreliable; do them one at a time.
- Radio and field **IDs are regenerated every time a wizard opens** (`radio-r8` becomes something
  else on the next run). Match on `input[type=radio].value` (which holds the output table name, e.g.
  `HighValueCustomers`) rather than on `id`.
- Combobox chevrons have **no `aria-label`** — reach them as the sibling `button` of the
  `input[placeholder="Select attribute"]` / `"Select values"`.

## ✅ Best way to create segments: build them FROM the suggestions

This supersedes the earlier advice to hand-build segments. Once the suggestions job has run, each
suggested RFM cell carries a **`Create segment`** button that opens a tiny *Name segment* dialog with
both fields **pre-filled and already valid**. No attribute picking, no operator, no threshold
guessing — and it sidesteps the flaky quick-segment builder entirely.

Caldova's run produced **22 suggestions** from a single activity, e.g.:

| Suggested cell | Members | Avg recency | Avg frequency | Avg monetary |
|---|---|---|---|---|
| High R · High F · High M ("champions") | 29 | 38 days | 9 transactions | 237 |
| High R · Medium F · High M | 12 | 37 days | 6 transactions | 206 |
| High R · Medium F · Low M | 7 | 40 days | 4 transactions | 44 |

Two were promoted to real segments in seconds: **`Champions high RFM`** and
**`Frequent low spenders`**.

### 🔴 Two gotchas in that little dialog

**1. Segment names reject hyphens.** Typing `Champions - high RFM` set the Name textbox to
`aria-invalid="true"` and **disabled Save with no visible error message**. Spaces are fine; the
hyphen is not. `Champions high RFM` validated instantly.

**2. The Save gate needs REAL typing.** Setting the value with a React value-setter and dispatching
`input`/`change` leaves **Save disabled** — the same stale-panel failure this file documents for the
Activities wizard. A genuine Playwright `fill()` on the textbox releases it every time.

> This is the practical rule for CI-Data wizards generally: **synthetic events populate the field but
> do not satisfy the validator.** If a Save/Next button stays greyed while the field visibly contains
> the right text, that is the cause — retype it for real.

Useful behaviour: **`Output table name` auto-derives from `Name`** as you type
(`Champions high RFM` → `ChampionshighRFM`), and it is immutable once saved.