Speech-enable external controls

External controls are controls that are outside the DOM of your speech-enabled page. For example, if your desktop app implements the Dragon Copilot SDK for JavaScript in a web view, external controls would be desktop-native controls in your app that are external to the web view.

External control API

The main API consists of the DragonCopilotSDK.dragon.customControls.external module and the DragonCopilotSDK.dragon.setFocusContext function. For more information, see the API reference.

DragonCopilotSDK.dragon.customControls.external contains the following members:

• add - A function to register an external control. Provide the current state of the control and details relevant for speech recognition.

  • id - A unique identifier by which both the Dragon Copilot SDK and your implementation should identify an external control.

  • containerId - The ID of the container/form/app.

  • text - The current text of the control.

  • selection - The character-wise, 0-based starting and ending indexes of the current selection of the control.

    start
end

  • status - The current speech-processing status of the control.

  • paragraph - The paragraph delimiter used in the control, for example "\n\n".

  • newLine - The line delimiter used in the control, for example "\n".

  • order - The ordinal number of the control among the speech-enabled external ones as it shows up on the UI.

• remove - A function to remove an external control. You need to provide the control's id.

• handle - A higher-order function which accepts a separate callback for the following commands:

  • text - replace text at a given selection.

  • selection - set selection.

  • focus - focus the given control.

  The registered callbacks are invoked after dictation. In these callbacks, you need to synchronize the provided values with the external control. Each registered handler is a handler for all external controls. The controlId parameter passed with the callback tells you which control the incoming command is addressed to.

• update - A function to signal a state change occurred to external controls.

  A signaled change doesn't have to contain all text, selection and focus information every time. However, when you call update() with information on a text change, you also need to pass the selection start and end values, so the cursor/selection change that occurs with the text change is also synchronized.

• reportError - A function to report that an error happened during the execution of a text command. Provide the same reqEventId parameter that you received in the 'handle text' command.

Set focus context

DragonCopilotSDK.dragon.setFocusContext switches focus context between web and external. If you speech-enable both web and external controls in your app, you need to call this when you switch between the external and web focus contexts.

Note

Timing is an extremely important factor when syncing dictation, speech recognition results, and UI states. It is especially important for custom external controls due to the signalling needed between web and external contexts. That is why every operation of this API passes a timestamp.

Example: Create a new external control

The sample code below demonstrates the following steps:

• Implement the 'signalling' layer between your external context and the Dragon Copilot SDK integration.

• Implement the command handlers.

• Add controls.

• Clean up the command handlers.

// Implement the 'signalling' layer between the external context and your integration.
// This would be completely specific to your case. Here we provide a 'dummy' example of the signalling layer, with no real implementation of it.
// We add only abstracts to the level that makes the other steps involved comprehensible.
interface ExternalTextChangedEvent {
  controlId: string;
  text: string;
  selection: { start: number; length: number };
  timestamp: number;
}

interface ExternalSelectionChangedEvent {
  controlId: string;
  selection: { start: number; length: number };
  timestamp: number;
}

interface ExternalFocusChangedEvent {
  controlId: string;
  focus: boolean;
  timestamp: number;
}
 
interface ExternalContextAbstract {
  fetchControls: () => Promise<DragonCopilotSDK.dragon.ExternalCustomControl[]>; // For the sake of simplicity of the example, we reuse the ExternalCustomControl type of the Dragon Copilot SDK.
  setText: (controlId: string, text: string, start: number, length: number, timestamp: number, reqEventId: string) => void;
  setSelection: (controlId: string, start: number, length: number, timestamp: number) => void;
  focus: (controlId: string, timestamp: number) => void;
  // Each event returns a cleanup function by which the listener can be unregistered.
  onControlAdded: (
    listener: (event: { control: DragonCopilotSDK.dragon.ExternalCustomControl; timestamp: number }) => void,
  ) => () => void; // For the sake of simplicity of the example, we reuse the ExternalCustomControl type of the Dragon Copilot SDK.
  onControlRemoved: (listener: (event: { controlId: string; timestamp: number }) => void) => () => void;
  onTextChanged: (listener: (event: ExternalTextChangedEvent) => void) => () => void;
  onSelectionChanged: (listener: (event: ExternalSelectionChangedEvent) => void) => () => void;
  onFocusChanged: (listener: (event: ExternalFocusChangedEvent) => void) => () => void;
}

const externalContext: ExternalContextAbstract = {
  // implementation of the abstract.
};

// This function binds the external context with the Dragon Copilot SDK.
async function setupForExternalControls(): Promise<() => void> {
  const cleanupCommandHandlers = registerCommandHandlers(externalContext);
  const cleanupExternalEventListeners = registerExternalEventListeners(externalContext);

  // Add the initial set of controls.
  const textControls = await externalContext.fetchControls();
  for (const control of textControls) {
    const isSuccess = DragonCopilotSDK.dragon.customControls.external.add(control, Date.now());
    if (!isSuccess) {
      console.error("Failed to add control with id:", control.id);
    }
  }

  return () => {
    cleanupCommandHandlers();
    cleanupExternalEventListeners();
  };
}

function registerCommandHandlers(signalAbstract: ExternalContextAbstract): () => void {
  // Note that each registration returns a cleanup function which can be called to unregister the handler when it is no longer needed.
  const cleanupTextCommandHandler = DragonCopilotSDK.dragon.customControls.external.handle(
    "text",
    ({ controlId, text, start, length, timestamp, reqEventId }) => {
      signalAbstract.setText(controlId, text, start, length, timestamp, reqEventId);
    },
  );

  const cleanupSelectionCommandHandler = DragonCopilotSDK.dragon.customControls.external.handle(
    "selection",
    ({ controlId, start, length, timestamp }) => {
      signalAbstract.setSelection(controlId, start, length, timestamp);
    },
  );

  const cleanupFocusCommandHandler = DragonCopilotSDK.dragon.customControls.external.handle("focus", ({ controlId, timestamp }) => {
    signalAbstract.focus(controlId, timestamp);
  });

  // Return cleanup of registered handlers.
  return () => {
    cleanupTextCommandHandler();
    cleanupSelectionCommandHandler();
    cleanupFocusCommandHandler();
  };
}

function registerExternalEventListeners(externalContext: ExternalContextAbstract): () => void {
  const cleanupControlAddedListener = externalContext.onControlAdded((event) => {
    DragonCopilotSDK.dragon.customControls.external.add(event.control, event.timestamp);
  });

  const cleanupControlRemovedListener = externalContext.onControlRemoved((event) => {
    DragonCopilotSDK.dragon.customControls.external.remove(event.controlId, event.timestamp);
  });

  // Note that you don't need to go this granular with the update.
  // If you'd like to have a single onControlChanged event defined by your external context, you can do that too.
  // Just set all text, start, length and focus in a single update call accordingly.
  const cleanupTextListener = externalContext.onTextChanged((event) => {
    DragonCopilotSDK.dragon.customControls.external.update(event.controlId, {
      text: event.text,
      start: event.selection.start,
      length: event.selection.length,
      timestamp: event.timestamp,
    });
  });

  const cleanupSelectionListener = externalContext.onSelectionChanged((event) => {
    DragonCopilotSDK.dragon.customControls.external.update(event.controlId, {
      start: event.selection.start,
      length: event.selection.length,
      timestamp: event.timestamp,
    });
  });
 
  const cleanupFocusListener = externalContext.onFocusChanged((event) => {
    DragonCopilotSDK.dragon.customControls.external.update(event.controlId, {
      focus: event.focus,
      timestamp: event.timestamp,
    });
  });

  return () => {
    cleanupControlAddedListener();
    cleanupControlRemovedListener();
    cleanupTextListener();
    cleanupSelectionListener();
    cleanupFocusListener();
  };
}

See also

API reference