v2.6.0

<bmx-data-profiler>

Data profiling and data quality in the page: drop in a CSV, TSV, JSON or Excel file (or hand it records) and see what is in every column and what is wrong with it - the first look an analyst, a data engineer, a data steward or an auditor takes at a dataset, and the check before a pipeline trusts one.

10 properties · 7 events · 8 methods · 3 parts

Example

Open the Columns tab to see each column; alerts on the Overview open the column they are about. Open a CSV, JSON or Excel file of your own with "Open a file…" - it is read in the page and never sent anywhere.

Show markup
<div role="group" aria-label="Sample data" style="display: flex; flex-wrap: wrap; gap: 8px; margin: 0 0 12px; align-items: center">
  <button type="button" id="ex-dp-baseline">Compare with last month</button>
  <button type="button" id="ex-dp-suggest">Suggest rules from the data</button>
</div>
<bmx-data-profiler id="ex-dp" name="Orders, September 2026" style="--bmx-data-profiler-height: 46rem"></bmx-data-profiler>
<p class="note" id="ex-dp-out" role="status" style="margin: 12px 0 0">Open the Columns tab to see each column; alerts on the Overview open the column they are about. Open a CSV, JSON or Excel file of your own with "Open a file…" - it is read in the page and never sent anywhere.</p>

<script type="module">
  await customElements.whenDefined('bmx-data-profiler');
  const profiler = document.getElementById('ex-dp');
  const out = document.getElementById('ex-dp-out');

  // A fictional shop's orders, made here with a seeded random generator - and given the faults real extracts have:
  // missing emails, a city typed three ways, stray spaces, a few impossible quantities, a delivery that took 45 days,
  // three dates next year, repeated rows, a constant column and a notes column that is mostly empty.
  const orders = (month, count, seed, shift) => {
    let s = seed;
    const rnd = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);
    const pick = list => list[Math.floor(rnd() * list.length)];
    const first = ['Amelia', 'Oliver', 'Isla', 'George', 'Ava', 'Noah', 'Freya', 'Arthur', 'Ivy', 'Leo', 'Rosie', 'Theo', 'Maya', 'Finn', 'Esme', 'Rory'];
    const last = ['Hughes', 'Patel', 'Morgan', 'Byrne', 'Khan', 'Walsh', 'Turner', 'Price', 'Doyle', 'Shah', 'Fraser', 'Lloyd'];
    const cities = [['London', 'SW1A 1AA'], ['Leeds', 'LS1 4AP'], ['York', 'YO1 7HH'], ['Bristol', 'BS1 4ST'], ['Glasgow', 'G1 1XQ'], ['Cardiff', 'CF10 1EP'], ['Norwich', 'NR2 1NH']];
    const products = [['Mug', 8.5], ['Teapot', 32], ['Plate set', 46], ['Bowl', 12], ['Jug', 24], ['Vase', 38], ['Tumbler', 9.5]];
    const rows = [];
    for (let i = 0; i < count; i++) {
      const [name, price] = pick(products);
      const qty = 1 + Math.floor(rnd() ** 2 * 6);
      const unit = +(price * shift.price * (0.9 + rnd() * 0.2)).toFixed(2);
      const discount = rnd() < 0.82 ? 0 : pick([0.05, 0.1, 0.15]);
      const total = +(qty * unit * (1 - discount)).toFixed(2);
      const [city, postcode] = pick(cities);
      const customer = `${pick(first)} ${pick(last)}`;
      const day = 1 + Math.floor(rnd() * 30);
      const channel = rnd() < shift.marketplace ? 'Marketplace' : pick(['Web', 'Web', 'Web', 'Phone', 'Store']);
      rows.push({
        order_id: `SO-${String(shift.start + i).padStart(6, '0')}`,
        ordered_at: `2026-${month}-${String(day).padStart(2, '0')} ${String(8 + Math.floor(rnd() * 12)).padStart(2, '0')}:${String(Math.floor(rnd() * 60)).padStart(2, '0')}`,
        customer,
        email: rnd() < shift.noEmail ? '' : `${customer.toLowerCase().replace(' ', '.')}${Math.floor(rnd() * 90) + 10}@example.com`,
        phone: rnd() < 0.3 ? '' : `07700 900${String(Math.floor(rnd() * 1000)).padStart(3, '0')}`,
        city: i % 97 === 5 ? city.toLowerCase() : i % 131 === 7 ? ` ${city} ` : i % 151 === 9 ? city.toUpperCase() : city,
        postcode,
        channel,
        product: name,
        quantity: i % 211 === 3 ? 'unknown' : i % 307 === 11 ? '-1' : String(qty),
        unit_price: unit.toFixed(2),
        discount: String(discount),
        total: total.toFixed(2),
        vat: (total / 6).toFixed(2),
        status: rnd() < shift.returns ? 'Returned' : pick(['Delivered', 'Delivered', 'Delivered', 'Shipped', 'Cancelled']),
        delivery_days: i % 173 === 4 ? '45' : String(1 + Math.floor(rnd() * 5)),
        gift_wrap: rnd() < 0.15 ? 'yes' : 'no',
        notes: rnd() < 0.85 ? '' : pick(['Leave with neighbour', 'Fragile - handle with care', 'Gift - no invoice in the box', 'Call on arrival', 'Back door please']),
        country: 'United Kingdom',
      });
    }
    // Three dates mistyped into next year, and a few rows exported twice.
    if (shift.faults) {
      for (const k of [17, 402, 911]) rows[k].ordered_at = rows[k].ordered_at.replace('2026', '2027');
      rows.push(rows[3], rows[60], rows[60], rows[777]);
    }
    return rows;
  };

  const current = orders('09', 1500, 20260930, { price: 1.08, marketplace: 0.18, noEmail: 0.07, returns: 0.12, start: 21001, faults: true });
  const lastMonth = orders('08', 1400, 20260831, { price: 1, marketplace: 0, noEmail: 0.02, returns: 0.06, start: 19601, faults: false });

  profiler.rules = [
    { kind: 'unique', column: 'order_id' },
    { kind: 'pattern', column: 'order_id', pattern: '^SO-\\d{6}$' },
    { kind: 'not-null', column: 'email', severity: 'warning' },
    { kind: 'type', column: 'email', type: 'email' },
    { kind: 'between', column: 'quantity', min: 1, max: 20 },
    { kind: 'not-future', column: 'ordered_at' },
    { kind: 'in-set', column: 'status', values: ['Delivered', 'Shipped', 'Returned', 'Cancelled'] },
    { kind: 'no-duplicate-rows' },
  ];
  profiler.data = current;

  document.getElementById('ex-dp-baseline').addEventListener('click', () => (profiler.baseline = lastMonth));
  document.getElementById('ex-dp-suggest').addEventListener('click', async () => {
    const rules = await profiler.suggest();
    profiler.view = 'rules';
    out.textContent = `suggest(): ${rules.length} rules the data meets as it stands - edit or delete any of them.`;
  });
  profiler.addEventListener('bmxProfile', e => (out.textContent = `bmxProfile: quality score ${e.detail.profile.score.overall}, ${e.detail.profile.alerts.length} alerts, in ${e.detail.profile.took} ms.`));
  profiler.addEventListener('bmxRulesRun', e => (out.textContent = `bmxRulesRun: ${e.detail.run.passed} passed, ${e.detail.run.failed} failed.`));
  profiler.addEventListener('bmxDrift', e => (out.textContent = `bmxDrift: ${e.detail.drift.summary.significant} columns shifted significantly, ${e.detail.drift.summary.added} added.`));
</script>

Overview a quality score and what it is made of, rows, columns, missing cells, duplicate rows, personal data, the types, and every alert ranked by severity. Columns each column's type and role, its statistics, histogram or timeline, most common values, patterns, the values that do not fit, outliers with their rows, the same value spelt different ways, and personal data (emails, phones, card and bank numbers). Correlations Pearson and Spearman between numbers, Cramér's V between categories, as a heatmap and the strongest pairs. Missing by column, where in the file, and what is missing together. Rules checks the data must pass (not null, unique, type, range, length, pattern, allowed values, not in the future, missing at most...), suggested from the data, edited in place, saved as JSON, checked with failing rows shown. Drift against a baseline file: schema changes, the population stability index and the Kolmogorov-Smirnov test, new and vanished categories, shifts in what is missing.

<bmx-data-profiler id="dp"></bmx-data-profiler>
<script>
  dp.data = await (await fetch('orders.csv')).text();   // or an array of records
  dp.addEventListener('bmxProfile', e => console.log(e.detail.profile.score.overall));
</script>

Everything runs in the page; nothing is sent anywhere. The engine is also a Node module (dist/node/bmx-data-profiler.mjs) for a pipeline, and the report exports as one HTML file.

Accessibility: views are tabs; the column list is a listbox with a search box; every chart is a picture with its figures in words and a table on request; alerts, pairs and drift rows are buttons that open what they are about; the rules are a form and a table.

Properties

PropertyAttributeTypeDefaultDescription
baseline property only unknown — An earlier extract to compare with, in any of the same forms. Mutable: a file opened with "Compare with…" sets it.
column column string — The column showing in the Columns view. Mutable.
data property only unknown — The data: CSV or TSV text, JSON text, an array of records or of arrays, or { columns, rows }.
label label string — The profiler's accessible name.
name name string — The dataset's name, shown at the top and used for exported files.
noFiles no-files boolean false Hide the file buttons (the page gives the data).
options property only BmxProfileOptions — Profiling options: thresholds, the date order, the sample for correlations.
rules rules readonly BmxProfileRule[] | Record<string, unknown> | string — The rules to check: an array, { rules } or JSON. Mutable: edits replace it.
strings strings Partial<BmxDataProfilerStrings> | string — Words to show instead of the English ones, as an object or JSON.
view view BmxDataProfilerView 'overview' The view showing. Mutable.

Events

EventDetailDescription
bmxColumnSelect { readonly column: string; } Fired when the Columns view shows another column.
bmxDrift BmxDataProfilerDriftDetail Fired when the data is compared with the baseline.
bmxError { readonly message: string; } Fired when data or a file cannot be read.
bmxExport BmxDataProfilerExportDetail Fired when an export is made from the Export menu. Cancel it to keep it from downloading.
bmxProfile BmxDataProfilerProfileDetail Fired when a profile is made.
bmxRulesChange BmxDataProfilerRulesChangeDetail Fired when the rules are edited, with all of them.
bmxRulesRun BmxDataProfilerRulesDetail Fired when the rules are checked.

Methods

MethodSignatureDescription
exportReport exportReport() => Promise<string> The report as one HTML page (with the rules and the drift, when there are any).
getDrift getDrift() => Promise<BmxProfileDrift | undefined> The latest comparison with the baseline.
getProfile getProfile() => Promise<BmxProfile | undefined> The latest profile.
getRuleRun getRuleRun() => Promise<BmxProfileRuleRun | undefined> The latest rules check.
loadFile loadFile(file: Blob & { name?: string; }, as?: "data" | "baseline") => Promise<void> Reads a file (CSV, TSV, TXT, JSON or XLSX) as the data, or as the baseline.
refresh refresh() => Promise<void> Profiles the data again.
showColumn showColumn(name: string) => Promise<void> Shows a column in the Columns view.
suggest suggest(apply?: boolean) => Promise<BmxProfileRule[]> Rules the data meets now, from its profile. They replace the rules when apply is true.

CSS shadow parts

PartDescription
header The title and the tools.
panel The view showing.
tabs The views.

CSS custom properties

PropertyDescription
--bmx-data-profiler-bar Bars of the charts.
--bmx-data-profiler-baseline Baseline bars in the drift view.
--bmx-data-profiler-height The profiler's height; the view scrolls inside it. Default: auto (as tall as its content).
--bmx-data-profiler-negative Negative correlations in the heatmap.
--bmx-data-profiler-positive Positive correlations in the heatmap.
--bmx-data-profiler-radius Corner radius of the profiler.