Language contract reference

This page is the domain reference for a role = "language" package. The package-definition API is documented separately in Plugin API. The names in the examples are the JSON and TypeScript names used by the plugin contract.

Language packages implement two operations:

AuthoredTrack -> plan -> PhonePlan
PhonePlan + timingEdits -> finalize -> EngineScore

The application supplies the authored track and timing edits. A language package owns language analysis, phone identity, phone timing, gap rows, and construction of the engine-specific score.

These contract types are exported by @amsvs/api; a package does not need a separate domain.ts just to type its plan() and finalize() arguments.

The engine accepts this complete set of 40 phoneme values:

a b br by ch cl d dy e f g gy h hy i j k ky m my n N ny o p pau py
r ry s sh sil t ts ty u v w y z

This is a broad engine vocabulary, not an UTAU recording-bank alias list. The old UTAU manual sampleprefix examples do not define this contract and should not be used as a restriction on language plugins. A plugin may still expose a smaller language-specific inventory, but every emitted phone or engine-row phoneme must be one of the values above.

pau, sil, and br are engine phonemes used for gaps and rests. They are not note lyrics and do not imply that the engine should be split into separate render calls.

plan(track) receives an amadeus.track/v2 track. Its complete shape is:

interface AuthoredTrack {
  schema: string;                 // "amadeus.track/v2"
  trackId: string;
  ppq: number;
  tempoMap: TempoPoint[];
  meterMap: MeterPoint[];
  extent: TrackExtent;
  languageRoute: string[];
  notes: AuthoredNote[];
  gaps: AuthoredGap[];
  extensions?: JsonObject;
}

interface TempoPoint { tick: number; bpm: number }
interface MeterPoint { tick: number; beats: number; beatType: number }
interface TrackExtent { startTick: number; endTick: number }

interface AuthoredNote {
  id: string;
  startTick: number;
  endTick: number;
  pitch: number;                  // MIDI note number
  lyric?: string;
  language?: string;
  ties?: string[];
  slurs?: string[];
  flags?: { breath?: boolean; accent?: boolean; staccato?: boolean };
  dynamic?: string;
  phoneOverride?: string[];
  pitchExpression?: PitchExpression;
  vibrato?: VibratoSettings;
  extensions?: JsonObject;
}

interface AuthoredGap {
  id: string;
  startTick: number;
  endTick: number;
  kind: "pau" | "sil" | "br";
  extensions?: JsonObject;
}

Tick fields are score coordinates. They are not the engine timing unit. The track’s tempoMap is needed when converting score positions to time.

A PhonePlan contains phones: PhonePlanPhone[] plus protocol and provenance:

interface PhonePlan {
  protocol: "amadeus.language/v2";
  trackSchema: "amadeus.track/v2";
  trackId: string;
  phones: PhonePlanPhone[];
  diagnostics: Diagnostic[];
  provenance: ModuleProvenance;
}

interface PhonePlanPhone {
  id: string;
  ownerId: string;
  sourceNoteIds?: string[];
  ownerKind: "note" | "gap";
  phone: string;
  role: "pre" | "anchor" | "tail" | "breath";
  weight: number;
  ghost: boolean;
  vacuum: boolean;
  velocity?: number;
  start: Time;
  end: Time;
  sourceIndex: number;
  sourceCount: number;
  tone: number;
  vowelSign: number;
  metadata: JsonValue;
  note: JsonValue;
  extensions?: JsonObject;
}

type JsonPrimitive = string | number | boolean | null;
type JsonValue = JsonPrimitive | JsonObject | JsonValue[];
interface JsonObject { readonly [key: string]: JsonValue }

interface PitchPoint {
  position: number; // 0..1 across the note
  cents: number;    // offset from written pitch
}

interface VibratoSettings {
  depthCents: number;
  rateHz: number;
  delay: number;    // fraction of note duration
  fadeIn: number;   // fraction of note duration
}

interface PitchExpression {
  points: PitchPoint[];
  vibrato?: VibratoSettings;
}

interface Diagnostic {
  severity: "info" | "warning" | "error";
  code: string;
  message: string;
  ownerId?: string;
  phoneId?: string;
}

interface ModuleProvenance {
  protocol: string;
  moduleId: string;
  moduleVersion: string;
  bundleHash?: string;
  selectedLanguage: string;
  route: string[];
}

id is the stable identity used by timing edits. Construct it from stable authored identities and deterministic source positions; do not use a random value or an array index that changes when unrelated phones are inserted. ownerId identifies the note or gap. sourceIndex and sourceCount describe the phone’s position among the phones derived from that owner.

Phone IDs must be unique. Each phone has a positive span, a known engine phone, a finite positive weight, and a valid source index. The plan may contain multiple phones for one note, including pre, anchor, and tail material. anchor is the normal editing target; breath is a distinct role for breath material.

There is no assumption that plugins use a particular inference frame size. The canonical timing unit at the language boundary is one hundred nanoseconds (100 ns). start100ns and end100ns are absolute positions from the start of the score, not durations and not 10 ms frame numbers. The host converts these times to model frames later.

The plugin-facing API also exposes these values as start and end; the suffixed names remain available for older plugins. Both spellings use the same Time value, and the host supplies the normalized aliases before invoking a plugin.

The timing edit shape is deliberately small:

interface TimingEdit {
  phoneId: string;
  boundaryOffset: Time;
}

boundaryOffset is a signed score-relative shift from the phone’s base boundary. It is not a replacement duration. The legacy boundaryOffset100ns spelling is also accepted. finalize applies edits to the identified phone boundaries, then emits final absolute phone/row spans. The resulting EngineScore has positive-duration rows and continuous global time: each row starts exactly where the previous row ends.

The API spelling is boundaryOffset; boundaryOffset100ns remains accepted for older plugins. A timing edit must provide at least one spelling; matching duplicates are allowed, while missing or conflicting values are rejected.

finalize returns an engine-specific score with this shape:

interface EngineScore {
  kind: "neutrino_sinsy_v1";
  protocol: "amadeus.language/v2";
  trackId: string;
  rows: EngineRow[];
  diagnostics: Diagnostic[];
  provenance: ModuleProvenance;
}

interface EngineRow {
  rowId: string;
  sourcePhoneId: string;
  segmentIndex: number;
  start: Time;
  end: Time;
  phoneme: string;
  noteId?: string;
  gapId?: string;
  scoreStartTick: number;
  scoreEndTick: number;
  midi: number | null;
  contexts?: ContextGroups;
}

`contexts` is optional for language engines that do not emit Sinsy full-context
labels. It must be present on every row when the score is used for the legacy
Sinsy full-context label export; otherwise that export is rejected explicitly.

Rows are ordered in global time. A phone may produce multiple rows; in that case segmentIndex starts at zero and increments without gaps for that sourcePhoneId. rowId is unique. sourcePhoneId must map back to a planned phone, and note rows carry noteId while gap rows carry gapId.

An EngineScore must use the documented kind and protocol, known phonemes, positive spans, continuous rows, unique row IDs, and contiguous segment numbering. When contexts is present, all context widths must be valid. Its trackId and provenance must match the phone plan.

Each row contains eleven context groups. Their exact widths are:

The serializer concatenates each separator immediately before its field value. Therefore the serialized full-context label is:

start100ns end100ns <serialized contexts>

The mono label uses the same absolute times and the row’s phoneme instead of the serialized contexts. Context values are opaque strings to a plugin; do not reorder, omit, or reinterpret fields, and do not use a UTAU alias as a substitute for this Sinsy context string.

Authored gaps are explicit AuthoredGap entries with score-tick spans and a kind of pau, sil, or br. Preserve each gap’s identity and map it to corresponding gap-owned phone-plan phones and engine rows.

The engine stream is continuous across notes and gaps. Gaps are ordinary model inputs, not separators between independent inference calls. Do not drop a rest because it has no MIDI note, and do not create a hole between adjacent rows. For gap rows, use midi: null, set gapId, and use the appropriate gap phoneme. For note rows, set noteId and the note MIDI pitch.

• Accept the full AuthoredTrack; do not reduce it to lyric strings.
• Emit only the documented 40 engine phonemes.
• Keep phone IDs stable across equivalent plan calls.
• Use absolute 100 ns values at plan/finalize boundaries.
• Treat timing edits as signed boundary shifts, not durations.
• Preserve gaps and produce one continuous EngineScore stream.
• Emit all 11 Sinsy groups with exactly their required field counts when providing legacy contexts.
• Keep sourcePhoneId, rowId, segmentIndex, note/gap ownership, and provenance internally consistent.