v3.0.0

<bmx-text-annotator>

Labels text for training and checking language models, in the page, with nothing sent anywhere. Spans may nest and overlap; relations between them are drawn as arcs; documents take classes and free-text answers; spans and attributes follow the page's own schema. Pre-labels from the customer's own model come in as suggestions with a confidence, to accept or reject one by one or all at once above a threshold. Labelling works from the keyboard alone: the caret moves by token, Shift widens the selection, a label's key applies it. Right-to-left and mixed text read in their own direction, and Chinese, Japanese, Korean and Thai are tokenized character by character. Long documents are drawn a block at a time as they scroll into view. Reads and writes JSONL (doccano), CoNLL (IOB1, IOB2, BIOES), BRAT standoff, spaCy JSON and its own project JSON. When several people label the same documents, each one's labels are kept as their layer: the agreement between them is measured (Cohen's and Fleiss' kappa, span F1), every disagreement is listed with who chose what, and a reviewer builds the adjudicated copy from them, then approves the document or sends it back with a note. The guidelines and every label's description and examples are in the panel.

15 properties · 6 events · 28 methods · 5 parts

Example

Try it with your own text and pre-labels in confidence: files are opened by your browser, on your device. Nothing is uploaded, no copy reaches us or anyone else, and nothing is kept once you close this tab.

Show markup
<bmx-text-annotator id="ex-textann" label="Product reviews - labelling round 3"></bmx-text-annotator>
<p class="note" id="ex-textann-out" role="status" style="margin: 12px 0 0"></p>

<bmx-text-annotator id="ex-textann-review" label="Support tickets - adjudication round 1" mode="review" author="Reviewer" style="margin-block-start: 24px; --bmx-text-annotator-height: 24rem"></bmx-text-annotator>
<p class="note" id="ex-textann-review-out" role="status" style="margin: 12px 0 0"></p>

<script type="module">
  await customElements.whenDefined('bmx-text-annotator');
  const el = document.getElementById('ex-textann');
  const out = document.getElementById('ex-textann-out');

  // Invented reviews of invented products. The spans marked with a confidence are pre-labels, as a team's own
  // model would suggest them, waiting to be accepted or rejected.
  el.schema = {
    entities: [
      { id: 'PRODUCT', name: 'Product', key: 'p', description: 'The thing being reviewed, by its name or model.', examples: ['Kestrel K2 kettle', 'Lumen Desk Lamp'] },
      { id: 'FEATURE', name: 'Feature', key: 'f', description: 'A part or quality of the product the reviewer judges. Include the words that say what it does.', examples: ['brushed steel body', 'boils a full litre in under three minutes'], attributes: [{ id: 'sentiment', label: 'Sentiment', options: ['good', 'bad', 'neutral'], required: true }] },
      { id: 'ORG', name: 'Company', key: 'c' },
      { id: 'PERSON', name: 'Person', key: 'n' },
      { id: 'PLACE', name: 'Place', key: 'l' },
    ],
    relations: [
      { id: 'has_feature', name: 'has', from: ['PRODUCT'], to: ['FEATURE'] },
      { id: 'made_by', name: 'made by', from: ['PRODUCT'], to: ['ORG'] },
      { id: 'works_for', name: 'works for', from: ['PERSON'], to: ['ORG'] },
    ],
    classes: [{ id: 'positive', name: 'Positive' }, { id: 'mixed', name: 'Mixed' }, { id: 'negative', name: 'Negative' }],
    classRequired: true,
    questions: [{ id: 'summary', text: 'The review in one line', required: true }],
    guidelines: [
      '# Reviews, round 3',
      'Label what a buyer would search for or complain about. Leave out words that only describe the reviewer.',
      '# Spans',
      '- Take the shortest text that names the thing: "Kestrel K2 kettle", not "The Kestrel K2 kettle from".',
      '- A company inside a product name is labelled twice: the product, and the company inside it.',
      '- Give every feature a sentiment.',
      '# Pre-labels',
      'The suggestions come from last round\'s model. Accept them only when the span is exactly right; otherwise reject and label it yourself.',
    ].join('\n'),
  };

  // Finds the nth occurrence of a phrase, so the offsets below stay readable.
  const at = (text, phrase, n = 0) => {
    let i = -1;
    for (let k = 0; k <= n; k++) i = text.indexOf(phrase, i + 1);
    return [i, i + phrase.length];
  };
  const doc = (id, name, text, spans, relations = [], extra = {}) => ({
    id,
    name,
    text,
    spans: spans.map(([phrase, label, more = {}], k) => {
      const [start, end] = at(text, phrase, more.nth ?? 0);
      return { id: `s${k + 1}`, start, end, label, ...more, nth: undefined };
    }),
    relations: relations.map(([from, to, label, more = {}], k) => ({ id: `r${k + 1}`, from, to, label, ...more })),
    ...extra,
  });

  const r1 = 'The Kestrel K2 kettle from Brightwater Home boils a full litre in under three minutes, and the brushed steel body stays cool to the touch. The lid hinge, though, started squeaking after a fortnight. Customer service in Leeds replaced it without fuss.';
  const r2 = 'Bought the Lumen Desk Lamp after reading a review by Priya Natarajan, who tests lighting for Halden Labs. The colour temperature dial is excellent; the USB port is too slow to charge a phone.';
  const r3 = 'Our office swapped to Fernway Notebooks. The paper takes fountain pen ink without bleeding, and the lay-flat binding is a joy.\nThe covers scuff quickly.';
  const r4 = 'اشتريت سماعات نوفا من شركة الريم في دبي. جودة الصوت ممتازة لكن البطارية ضعيفة.';
  const r5 = '東京の店でアオバ社の炊飯器を買った。ご飯はとても美味しいが、操作パネルが分かりにくい。';
  const longText = Array.from({ length: 40 }, (_, i) =>
    `Week ${i + 1}: the Tessel Robot Vacuum from Corvid Appliances mapped the flat in one pass. Suction on rugs stayed strong, the bin needed emptying daily, and the app ${i % 3 ? 'kept the schedule' : 'lost the schedule once'}.`,
  ).join('\n');

  el.documents = [
    doc('rev-101', 'Kestrel K2 kettle', r1,
      [
        ['Kestrel K2 kettle', 'PRODUCT'],
        ['Brightwater Home', 'ORG'],
        ['boils a full litre in under three minutes', 'FEATURE', { attributes: { sentiment: 'good' } }],
        ['brushed steel body', 'FEATURE', { confidence: 0.91 }],
        ['lid hinge', 'FEATURE', { confidence: 0.84 }],
        ['Leeds', 'PLACE', { confidence: 0.97 }],
        ['Customer service', 'ORG', { confidence: 0.41 }],
      ],
      [['s1', 's2', 'made_by'], ['s1', 's3', 'has_feature'], ['s1', 's4', 'has_feature', { confidence: 0.77 }]],
      { suggestedClasses: [{ id: 'mixed', confidence: 0.71 }] }),
    doc('rev-102', 'Lumen Desk Lamp', r2,
      [['Lumen Desk Lamp', 'PRODUCT'], ['Priya Natarajan', 'PERSON', { confidence: 0.95 }], ['Halden Labs', 'ORG', { confidence: 0.93 }], ['colour temperature dial', 'FEATURE', { confidence: 0.88 }], ['USB port', 'FEATURE', { confidence: 0.86 }]],
      [['s2', 's3', 'works_for', { confidence: 0.82 }]]),
    doc('rev-103', 'Fernway Notebooks', r3, [['Fernway Notebooks', 'PRODUCT']]),
    doc('rev-104', 'Nova earphones (Arabic)', r4, [['سماعات نوفا', 'PRODUCT', { confidence: 0.8 }], ['الريم', 'ORG', { confidence: 0.75 }], ['دبي', 'PLACE', { confidence: 0.9 }]]),
    doc('rev-105', 'Aoba rice cooker (Japanese)', r5, [['東京', 'PLACE', { confidence: 0.92 }], ['アオバ社', 'ORG', { confidence: 0.7 }], ['炊飯器', 'PRODUCT', { confidence: 0.66 }]]),
    doc('rev-106', 'Tessel vacuum, 40-week diary', longText, [['Tessel Robot Vacuum', 'PRODUCT'], ['Corvid Appliances', 'ORG']], [['s1', 's2', 'made_by']]),
  ];

  const say = text => (out.textContent = text);
  el.addEventListener('bmxChange', e => {
    const d = e.detail.document;
    const live = d.spans.filter(s => s.suggestion !== 'rejected' && s.suggestion !== 'suggested').length;
    say(`${d.name}: ${e.detail.reason.replace('-', ' ')}, ${live} spans labelled, status ${d.status}.`);
  });
  el.addEventListener('bmxDocumentChange', e => say(`Showing ${e.detail.id}.`));

  // A reviewer's view: three invented annotators labelled the same support tickets. The Review tab lists where they
  // differ, with who chose what; the Agreement tab measures how far they agree.
  const rv = document.getElementById('ex-textann-review');
  const rvOut = document.getElementById('ex-textann-review-out');
  rv.schema = {
    entities: [{ id: 'PRODUCT', name: 'Product' }, { id: 'ISSUE', name: 'Issue' }, { id: 'VERSION', name: 'Version' }],
    relations: [{ id: 'affects', name: 'affects', from: ['ISSUE'], to: ['PRODUCT'] }],
    classes: [{ id: 'bug', name: 'Bug' }, { id: 'question', name: 'Question' }, { id: 'request', name: 'Feature request' }],
    guidelines: 'Label the product, its version and the issue in the customer\'s own words. One class per ticket.',
  };
  const ticket = (id, text, labels, classes, status) => ({
    id,
    text,
    status,
    layers: Object.entries(labels).map(([annotator, spans]) => ({
      annotator,
      spans: spans.map(([phrase, label]) => {
        const [start, end] = at(text, phrase);
        return { start, end, label };
      }),
      classes: [classes[annotator]],
    })),
  });
  rv.documents = [
    ticket('T-4410', 'Harbourline Sync 3.2 crashes when a folder name has an emoji in it.',
      { Asha: [['Harbourline Sync', 'PRODUCT'], ['3.2', 'VERSION'], ['crashes', 'ISSUE']], Ben: [['Harbourline Sync', 'PRODUCT'], ['3.2', 'VERSION'], ['crashes when a folder name has an emoji in it', 'ISSUE']], Chen: [['Harbourline Sync 3.2', 'PRODUCT'], ['crashes', 'ISSUE']] },
      { Asha: 'bug', Ben: 'bug', Chen: 'bug' }, 'todo'),
    ticket('T-4411', 'Can Quillmark export to EPUB, or only PDF?',
      { Asha: [['Quillmark', 'PRODUCT']], Ben: [['Quillmark', 'PRODUCT'], ['export to EPUB', 'ISSUE']], Chen: [['Quillmark', 'PRODUCT']] },
      { Asha: 'question', Ben: 'request', Chen: 'question' }, 'todo'),
    ticket('T-4412', 'Please add dark mode to the Tallyhouse mobile app.',
      { Asha: [['Tallyhouse mobile app', 'PRODUCT'], ['dark mode', 'ISSUE']], Ben: [['Tallyhouse mobile app', 'PRODUCT'], ['dark mode', 'ISSUE']], Chen: [['Tallyhouse', 'PRODUCT'], ['dark mode', 'ISSUE']] },
      { Asha: 'request', Ben: 'request', Chen: 'request' }, 'todo'),
  ];
  rv.addEventListener('bmxReview', e => (rvOut.textContent = `${e.detail.document.id}: ${e.detail.status}${e.detail.note ? ` - ${e.detail.note}` : ''}.`));
  rv.addEventListener('bmxChange', e => {
    if (e.detail.reason === 'adjudicate') rvOut.textContent = `${e.detail.document.id}: gold copy has ${e.detail.document.spans.length} spans.`;
  });
</script>
<bmx-text-annotator src="/data/reviews.jsonl"
  schema='{"entities":["PRODUCT","FEATURE"],"relations":["has"],"classes":["positive","negative"]}'></bmx-text-annotator>

Properties

PropertyAttributeTypeDefaultDescription
author author string — Written as the author of every span and relation made here.
current current string — The id of the document shown.
documents documents readonly unknown[] | string | null — The documents alone, as objects (text, spans, relations, classes …), plain strings, or JSON; they replace the project's.
guidelines guidelines string — The labelling guidelines as text (# starts a heading line, - a bullet), in place of the schema's.
label label string — A name for the annotator, shown in the toolbar and given to assistive technology.
mode mode BmxTextAnnotatorMode 'annotate' annotate, or review (approve documents or send them back, with a note).
panel panel boolean true Shows the side panel.
project project BmxTextAnnotationProject | Record<string, unknown> | string | null — The project: the schema and the documents, as an object or JSON.
readonly readonly boolean false Shows the labels without letting them change.
schema schema BmxTextAnnotationSchema | Record<string, unknown> | string | null — The schema alone (entity labels, relation labels, classes, questions), as an object or JSON; it replaces the project's.
snap snap BmxTextAnnotatorSnap 'token' How a mouse selection is grown: token (to whole tokens) or character.
src src string — A file to open: JSONL, CoNLL, BRAT .ann (with its .txt beside it), spaCy JSON, project JSON or plain text.
strings strings Partial<BmxTextAnnotatorStrings> | string — Words to show instead of the English ones, as an object or JSON.
threshold threshold number 0.8 The confidence at or above which "Accept at or above" accepts suggestions.
toolbar toolbar boolean true Shows the toolbar.

Events

EventDetailDescription
bmxChange { readonly document: BmxTextAnnotationDocument; readonly reason: BmxTextAnnotatorChange; } After a span, relation or class changes; document is the document as it now is.
bmxDocumentChange { readonly id: string; readonly index: number; } After another document is shown.
bmxError { readonly message: string; } When a file cannot be read or written.
bmxLoad { readonly format: string; readonly documents: number; readonly notes: readonly string[]; } After files or src are opened.
bmxReview { readonly document: BmxTextAnnotationDocument; readonly status: BmxTextAnnotationStatus; readonly note: string; } After a document is approved or sent back.
bmxSelect { readonly span: BmxTextAnnotationSpan | null; readonly relation: BmxTextAnnotationRelation | null; } After the selected span or relation changes.

Methods

MethodSignatureDescription
accept accept(id?: string) => Promise<boolean> Accepts a suggestion (the selected one by default).
acceptAbove acceptAbove(threshold?: number) => Promise<number> Accepts every suggestion in the shown document at or above a confidence (default: threshold); returns how many.
addAnnotator addAnnotator(name: string, source: File | Blob | FileList | readonly (File | Blob)[] | string | BmxTextAnnotationProject | Record<string, unknown>, fileName?: string) => Promise<{ matched: number; added: number; }> Adds one annotator's labels as their layer of each document (matched by id, else by the same text; new documents are added): files of any format the annotator reads, text with a file name to tell its format, or a project.
addRelation addRelation(from: string, to: string, label: string) => Promise<string> Joins two spans of the shown document; returns the relation's id, or '' when the schema does not allow it.
addSpan addSpan(start: number, end: number, label: string, extra?: Partial<Pick<BmxTextAnnotationSpan, "attributes" | "note" | "confidence" | "suggestion" | "author">>) => Promise<string> Labels text in the shown document (UTF-16 offsets); returns the new span's id, or '' when nothing was added.
adjudicate adjudicate(how?: "majority" | "unanimous" | string) => Promise<{ decided: number; undecided: number; }> Builds the shown document's gold copy: every majority choice, every unanimous one, or one annotator's labels (by name).
check check() => Promise<BmxTextAnnotationProblem[]> Everything wrong with every document, measured against the schema.
export export(format?: BmxTextAnnotatorExport, options?: BmxTextAnnotationExportOptions) => Promise<Blob> The labels as a file: project, jsonl, conll (IOB2), conll-bioes, brat (a ZIP of .txt and .ann) or spacy.
getAgreement getAgreement(options?: BmxTextAnnotationAgreementOptions) => Promise<BmxTextAnnotationAgreement> How far the annotators agree: Cohen's and Fleiss' kappa over tokens and classes, span and relation F1, per label, and each against the gold copy.
getAnnotators getAnnotators() => Promise<string[]> The annotators, in the order they first appear.
getDisagreements getDisagreements(all?: boolean) => Promise<BmxTextAnnotationDisagreement[]> The places in the shown document where the annotators differ (with all, also where they agree).
getProject getProject() => Promise<BmxTextAnnotationProject> The project as it now is: the schema and every document.
load load(source: File | Blob | FileList | readonly (File | Blob)[] | string, name?: string) => Promise<void> Opens files of any format the annotator reads: File, Blob, FileList, an array of them, or text with a file name to tell its format.
next next() => Promise<void> Shows the next document.
previous previous() => Promise<void> Shows the previous document.
redo redo() => Promise<boolean> Redoes the last change undone.
reject reject(id?: string) => Promise<boolean> Rejects a suggestion (the selected one by default).
rejectWaiting rejectWaiting() => Promise<number> Rejects every suggestion in the shown document still waiting; returns how many.
removeAnnotator removeAnnotator(name: string) => Promise<void> Takes an annotator's layer out of every document.
removeRelation removeRelation(id: string) => Promise<boolean> Removes a relation.
removeSpan removeSpan(id: string) => Promise<boolean> Removes a span and its relations.
review review(status: BmxTextAnnotationStatus, note?: string) => Promise<void> Approves the shown document, sends it back (rejected) or sets another status, with the reviewer's note.
save save(format?: BmxTextAnnotatorExport, fileName?: string) => Promise<void> Downloads the labels as a file (see export).
select select(id: string | null) => Promise<void> Selects a span or relation of the shown document, or nothing.
selectDocument selectDocument(document: string | number) => Promise<void> Shows a document, by id or by its place (from 0).
setProject setProject(project: BmxTextAnnotationProject | Record<string, unknown> | string) => Promise<void> Shows a project (an object or JSON), starting the undo history again.
showLayer showLayer(annotator: string | null) => Promise<void> Shows an annotator's labels (read only), or with null the document's own (the gold copy).
undo undo() => Promise<boolean> Undoes the last change.

CSS shadow parts

PartDescription
labels The bar of labels.
panel The side panel (document, spans, selected, documents).
status The line under the text.
text The text being labelled.
toolbar The bar of commands.

CSS custom properties

PropertyDescription
--bmx-text-annotator-font-size The size of the text being labelled.
--bmx-text-annotator-height The height of the text and panel.
--bmx-text-annotator-panel-width The side panel's width.
--bmx-text-annotator-radius Corner radius of the frame.