Escaping of LaTeX delimiters is overly aggressive

[dlqqq]

Problem

Because Jupyter AI uses the JupyterLab renderer for Markdown + LaTeX, we need to add an extra backslash to LaTeX delimiters so math expressions are rendered when they are used.

Our current logic is very simple and directly escapes the message body as a string before passing it to the renderer:

/**
 * Escapes backslashes in LaTeX delimiters such that they appear in the DOM
 * after the initial MarkDown render. For example, this function takes '\(` and
 * returns `\\(`.
 *
 * Required for proper rendering of MarkDown + LaTeX markup in the chat by
 * `ILatexTypesetter`.
 */
function escapeLatexDelimiters(text: string) {
  return text
    .replace(/\\\(/g, '\\\\(')
    .replace(/\\\)/g, '\\\\)')
    .replace(/\\\[/g, '\\\\[')
    .replace(/\\\]/g, '\\\\]');
}

However, sometimes the LLM provides a response that literally includes \(, \[, \], or \). In other words, there are scenarios where these symbols appear but do not serve as TeX delimiters, and therefore should not be escaped. Currently, when this happens, an extra backslash is erroneously added.

In the below example, the UI wrongly shows three backslashes in the negative lookbehind instead of two. The frontend is transforming ?<![$\\]) into ?<![$\\\]), which is invalid.

Proposed Solution

Do not escape LaTeX delimiters if:

1. The delimiter is already escaped (i.e. escapeLatexDelimiters('\\(') should return '\\('), or
2. The delimiter is within MarkDown inline or block code.

The second step will be trickier, since I'm not sure if it can be handled exclusively via a regex. This may have to be done imperatively. We should also explore if there is a third-party package that provides an implementation of this.