Home → Questions →

Marked2 refresh breaks with pandoc/katex

• Edit

Steven Obua

08 Sep, 2020 01:01 PM

Hi,

I am using a custom preprocessor (pandoc) with the following options:

• Path: /usr/local/bin/pandoc
• Args: -f markdown -t html5 -s --katex

This leads to the refresh not working properly anymore. That is, when I change the .md file, the inline math is not displayed properly in Marked 2. Explicitly right-clicking and selecting "Refresh" in the Marked2 window fixes it, but that's obviously not a great workflow.

I am using Marked 2 V2.6.1

1. Support Staff 1 Posted by Brett on 08 Sep, 2020 01:45 PM

   

   Remove the —katex flag and let me know if that helps.

   - Brett

   • Edit

2. 2 Posted by Steven Obua on 08 Sep, 2020 02:24 PM

   

   Hi Brett,

   thank you for looking into this. Removing the --katex flag fixes the refresh problem.

   Unfortunately I need the --katex flag to force pandoc to use proper latex output instead of unicode. I guess if pandoc had a flag --dontUseUnicode my problem would be solved.

   • Edit

3. Support Staff 3 Posted by Brett on 08 Sep, 2020 02:34 PM

   

   I'm not familiar enough with Pandoc to be certain why the katex flag is
   messing up the refresh to begin with. It's odd that it works on first
   load (or refresh) but not on update. I don't currently have an
   explanation for that.

   -Brett

   • Edit

4. 4 Posted by Steven Obua on 08 Sep, 2020 02:41 PM

   

   No worries, I'll find a workaround for that. I think the problem is somehow that pandoc includes a link to the katex-script with the --katex option.

   • Edit

5. Support Staff 5 Posted by Brett on 08 Sep, 2020 02:46 PM

   

   Yeah, that would botch Marked's built-in script as it loads a
   conflicting script with every refresh… I might be able to test for the
   existence of an embedded script and compensate for it, but I'll have to
   look into it.

   -Brett

   • Edit

6. Support Staff 6 Posted by Brett on 08 Sep, 2020 05:02 PM

   

   So without the --katex flag, Pandoc appears to do its own mathML
   rendering, but with the flag it simply outputs a span with the class
   "math inline". Which _should_ be exactly what the built-in katex
   processor needs to do its own work, so it appears it's working in
   reverse. I'm presently very confused, but will continue looking into
   this as you're the third person to report the issue with the exact same
   circumstances.

   As far as the unicode processing issue, can you provide me with a sample
   document that shows the type of input you're giving it and getting
   incorrect output when `--katex` isn't included in the arguments? (Not
   being a "math" guy myself, I might need an explanation of what's
   incorrect about the rendering, but we'll see :)).

   Thanks,
   Brett

   • Edit

7. 7 Posted by Steven Obua on 08 Sep, 2020 06:20 PM

   

   Yes, I see a little bit clearer now myself. I've been using Pandoc seriously just since yesterday, so just getting used to its features myself.

   Here is an example markdown input: $c \in \mathcal{C}$

   You are right, without the --katex flag, Pandoc is doing its own rendering using built-in features of the browser, producing output like
   

   <span class="math inline"><em>c</em> ∈ 풞</span>
   

   This works with Marked2 without problems.

   With the --katex flag, it produces output meant for post processing with the Katex script, like this:

   <span class="math inline">\(c \in \mathcal{C}\)</span>
   

   It seems to me almost that the problem is that the post-processing that is supposed to happen via the KaTeX engine doesn't happen on automatic refresh somehow, but just on manual refresh. Are you actually loading the site new on automatic refresh, or are you doing something maybe that updates the page without triggering the KaTeX postprocessing of the DOM?

   • Edit

8. Support Staff 8 Posted by Brett on 08 Sep, 2020 06:43 PM

   

   Automatic updates happen by injecting new content into the DOM rather than a full refresh (as of version 1.6). However, on every update it should be triggering MathJax/Katex rendering (depending on what’s enabled in preferences). So I just need to figure out why this works with the built-in processors but not with pandoc…

   - Brett

   • Edit

9. 9 Posted by Steven Obua on 08 Sep, 2020 08:46 PM

   

   As far as I can tell, Pandoc doesn't do anything special except insert the inline-math as usual and then inserting code in the html header to load the script. So I guess the difference must be in how the script loading works in the various circumstances.

   I have Katex rendering enabled in Marked2 preferences, if that makes a difference?

   With the standard Pandoc installation this header is generated:

   <script src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.8.3/katex.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.8.3/contrib/auto-render.min.js"></script><script>document.addEventListener("DOMContentLoaded", function() {
   renderMathInElement(document.body);
   });</script><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.8.3/katex.min.css" />
   

   This leads to the observed behaviour. I've also tried changing the template that Pandoc uses so that instead the following header is generated to include the newest katex:

   <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/katex.min.css" integrity="sha384-AfEj0r4/OFrOo5t7NnNe46zW/tFgW6x/bCJG8FqQCEo3+Aro6EYUG4+cU+KJWu/X" crossorigin="anonymous">
   
   <!-- The loading of KaTeX is deferred to speed up page rendering -->
   <script defer src="https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/katex.min.js" integrity="sha384-g7c+Jr9ZivxKLnZTDUhnkOnsh30B4H0rpLUpJ4jAIKs4fnJI+sEnkvrMWph2EDg4" crossorigin="anonymous"></script>
   
   <!-- To automatically render math in text elements, include the auto-render extension: -->
   <script defer src="https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/contrib/auto-render.min.js" integrity="sha384-mll67QQFJfxn0IYznZYonOWZ644AWYC+Pt2cHqMaRhXVrursRwvLnLaebdGIlYNa" crossorigin="anonymous"
       onload="renderMathInElement(document.body);"></script>
   

   Doesn't work either, even worse: now even manual refresh doesn't work anymore.

   When I switch off custom processing, the update works without problems. Unfortunately I cannot use it as I am relying on some pandoc features like the ability to define new latex macros.

   I've looked at the html code that is exported by Marked 2, and adapted the pandoc template so that it looks the same. No difference, still doesn't work.

   • Edit

10. Support Staff 10 Posted by Brett on 08 Sep, 2020 09:11 PM

    

    So the scripts you show are running renderMathInElement only on
    DOMContentLoaded events, which wouldn't be triggered by the update.
    However, I note that manually running renderMathInElement doesn't work
    with the markup that Pandoc is inserting when `--katex` is specified.
    Something is missing from that resulting markup that isn't allowing
    Katex to perform its rendering, whether loaded by Marked or by Pandoc.
    Still investigating...

    -Brett

    • Edit

11. Support Staff 11 Posted by Brett on 08 Sep, 2020 09:27 PM

    

    Ok, so test this out for me, if you don't mind:

    1. Include the --katex flag in the custom processor arguments
    2. Load a file where the math no longer renders now
    3. Right click in the preview and choose "Inspect Element"
    4. In the inspector, click in the bottom (the "Terminal") and paste in
    the following:

    document.querySelectorAll('.math').forEach(function(i) {
    katex.render(i.innerText, i); })

    See if that causes everything to render the way you expect it to. If so,
    I'll just need to figure out a way to trigger that when needed... which
    will probably be just as confusing, but at least I'll have a start.

    -Brett

    • Edit

12. 12 Posted by Steven Obua on 08 Sep, 2020 09:45 PM

    

    Of course, happy to test!

    I've tried it, and it looks much better, the inline math is correct after running the code. But the display math (the one with [ ... ]) is shown inline instead of in display mode. That's probably because not only the display math element itself must be updated, but also the paragraph it is in.

    • Edit

13. 13 Posted by Steven Obua on 08 Sep, 2020 10:19 PM

    

    I think the problem is that somehow the \( \) or \[ \] that surround the math text get lost. If in the terminal I execute

    document.querySelectorAll('.math').forEach(function(i) { 
        i.innerText = "\\[" + i.innerText + "\\]";});
    

    and then

    renderMathInElement(document.body);
    

    every math element is shown in display mode.

    • Edit

14. 14 Posted by Steven Obua on 08 Sep, 2020 10:57 PM

    

    So if I type this into the terminal, everything displays perfectly:

    document.querySelectorAll('.math.inline').forEach(function(i) { 
        i.innerText = "\\(" + i.innerText + "\\)";});
    document.querySelectorAll('.math.display').forEach(function(i) { 
        i.innerText = "\\[" + i.innerText + "\\]";});
    renderMathInElement(document.body);
    

    But instead of running that code it's probably better to make sure that the escaped brackets don't get lost in the first place.

    • Edit

15. Support Staff 15 Posted by Brett on 08 Sep, 2020 11:05 PM

    

    Yes, I was noticing that the delimiters were stripped by pandoc when —katex is used. Will have to work around that.

    - Brett

    • Edit

16. 16 Posted by Steven Obua on 08 Sep, 2020 11:24 PM

    

    Oh, but they are not stripped by pandoc! KaTex does not work when the delimiters are not there. When I redirect the pandoc output into a file, they are there. So its probably happening when you take the input from the custom processor and do your thing with it.

    • Edit

17. Support Staff 17 Posted by Brett on 08 Sep, 2020 11:29 PM

    

    Interesting, I’ll look into that.

    - Brett

    • Edit

18. Support Staff 18 Posted by Brett on 09 Sep, 2020 02:21 AM

    

    When I run pandoc --katex on the following (from the command line):

    $\Delta\delta = \delta_2 - \delta_1$
    
    \\(\Delta\delta = \delta_2 - \delta_1\\)
    

    I get:

    <p><span class="math inline">\Delta\delta = \delta_2 - \delta_1</span></p>
    <p>\(= _2 - _1\)</p>
    

    Using $ delimiters, it creates the math span but removes the delimters. Using the escaped parens preserves the delimiters, but removes the \deltas. Neither renders properly. This appears to be a pandoc issue, though I can't understand how it's supposed to function.

    • Edit

19. 19 Posted by Steven Obua on 09 Sep, 2020 05:12 AM

    

    Mysterious stuff!! I am trying the same input. I've attached the file test.md which I run it on, and then I run pandoc -f markdown -t html5 --katex test.md and I get the output

    <p><span class="math inline">\(\Delta\delta = \delta_2 - \delta_1\)</span></p>
    <p>\(= _2 - _1\)</p>
    

    So what is happening here is that the first one works OK, and the second one is probably just not valid pandoc math syntax?

    • test.md 79 Bytes
    • Edit

20. 20 Posted by Steven Obua on 09 Sep, 2020 06:57 AM

    

    I just checked and the missing brackets is definitely a pandoc issue. I was using a different version on the command line vs. in Marked 2. On the command line I was using the pandoc from my anaconda3 installation which was version 2.2.3.2. In Marked 2 I was using 2.10.1.

    The newer version omits the escaped brackets and that breaks the current version of KaTeX.

    • Edit

21. Support Staff 21 Posted by Brett on 09 Sep, 2020 07:02 AM

    

    I wonder if it's intentional somehow. Like did KaTeX change the way it
    operates in a newer version or something?

    -Brett

    • Edit

22. 22 Posted by Steven Obua on 09 Sep, 2020 07:16 AM

    

    In the newer version they are now doing this to load katex:

    <script src="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/katex.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function () {
     var mathElements = document.getElementsByClassName("math");
     var macros = [];
     for (var i = 0; i < mathElements.length; i++) {
      var texText = mathElements[i].firstChild;
      if (mathElements[i].tagName == "SPAN") {
       katex.render(texText.data, mathElements[i], {
        displayMode: mathElements[i].classList.contains('display'),
        throwOnError: false,
        macros: macros,
        fleqn: false
       });
    }}});
    </script>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/katex.min.css" />
    

    So that explains why they are not doing the escaped brackets anymore, because they are calling katex.render manually now.

    • Edit

23. 23 Posted by Steven Obua on 09 Sep, 2020 07:53 AM

    

    Ok, what I do now is: I use the

    • --mathjax flag for pandoc
    • switch on Mathjax in the Marked2 preferences

    Now it works flawlessly 

    I thought I tried that earlier, but probably it didn't work because of some combination of the changes I tried out with the wrong pandoc version and my custom pandoc template modifications.

    • Edit

24. 24 Posted by Steven Obua on 09 Sep, 2020 08:01 AM

    

    Oh, I was too quick, now there are some other issues. I don't think it is worth for you to invest any more time into that, I guess it is just a can of worms. I will just use the built-in rendering without --mathjax or --katex flag which is good enough for prototyping and just check the result from time to time in the browser with --katex switched on.

    Thank you so much for your help!

    • Edit

25. Support Staff 25 Posted by Brett on 10 Sep, 2020 08:15 AM

    

    I have a build that solves a lot of the issues, though there's still some weirdness with the way Pandoc strips out some characters when --katex is enabled. But give this a test and see if it helps overall:

    Marked 2.6.2 (1008)

    Thanks,
    Brett

    • Edit

26. 26 Posted by Steven Obua on 10 Sep, 2020 09:18 AM

    

    I see you are not one to easily give up!

    Here are my observations for the new version:

    • when using --katex and Katex flag in Marked2 preferences on, the update now works, and scrolling to the right position also works. But it is garbled up, it seems that macros are lost, for example in $(c_1, \ldots, c_n)$ the c_1 and c_2 render fine, but the \ldots is missing.

    • when using --mathjax and Mathjax flag on, update works, but it scrolls always to the top of the document. Also subindices do not work, as in $S_1$. There are some minor other problems as well.

    • when using no -- option and neither Mathjax flag or Katex flag on, everything works great except for the fact that the built-in browser renderers are so atrocious, and display formulas appear inline and not display. When the Katex flag is on, it is all garbled up though, and when the Mathjax flag is on the subindices problem appears.

    • Edit

27. Support Staff 27 Posted by Brett on 10 Sep, 2020 01:20 PM

    

    Ok, please try the same tests with this version. I think we're making progress! (I hope.)

    Marked 2.6.2 (1009)

    • Edit

28. Support Staff 28 Posted by Brett on 10 Sep, 2020 01:37 PM

    

    On your second point, the subindices were breaking because I was
    manually escaping underscores, which apparently I don't need to do? I
    know there was a reason for it at some point, but I've lost track. As
    far as the scrolling goes when MathJax is enabled, I am seeing it scroll
    up a bit on refresh, but if "Scroll to first Edit" is enabled in
    Preferences->Preview, it restores its position properly. I'm not certain
    why MathJax is causing that and KaTeX isn't, will look into it.

    Let me know how the latest test build goes.

    -Brett

    • Edit

29. 29 Posted by Steven Obua on 10 Sep, 2020 02:26 PM

    

    It works much much better now! As far as I can see, the only problem left is the scrolling to the top, which for me happens when the "Scroll to first Edit" is enabled. The edit-marker is also displayed at the very top.

    • Edit

30. 30 Posted by Steven Obua on 10 Sep, 2020 02:32 PM

    

    I had provided pandoc with a custom template with --template custom.html, and when I disable this, then "Scroll to first Edit" works properly! That's fine for me, as I don't need the custom template for editing, just later for publishing.

    Thank you for digging into this and making it work!

    • Edit