Lessons from the Trenches: Building a Scholarly Editor from Scratch

For years, our goal of pioneering HTML-first publishing workflows faced a major roadblock – the lack of robust online editors capable of maintaining high fidelity source integrity throughout the workflow. Existing web tools often fell short, lacking the fidelity and control essential for publishing scenarios.

After an arduous developmental journey, we created the Wax Editor to solve this problem. Built to empower HTML-first workflows while retaining publishing-grade fidelity, Wax now features across all Coko products as our flagship editor. But arriving at this solution involved navigating countless unforeseen obstacles.

This is a brief reflection on what it means to strive for innovation in the publishing industry with an HTML-first approach, at a time when the essential tools for such an endeavor were largely nonexistent.

Try 1: Building for PLOS and with the Wikipedia Foundation

In 2012, or so, I was building a new kind of submission and review platform for PLOS. The platform would manage content production using HTML as the underlying format. This led us to collaborate with the Wikimedia Foundation to enhance the, then, embryonic Wikipedia Visual Editor.

There wasn't anything as sophisticated at the time, it was, in effect, the only choice. However, the Wikipedia Visual Editor, being pioneering in its own right, was primarily focused on translating to and from wiki markup. This focus resulted in a codebase that was complex and not entirely suited to the needs of scholarly publishing. The visual editor project, although groundbreaking, turned out to be both time-consuming and costly for us.

More importantly, it shed light on the necessity for a dedicated "publishing grade" editor designed for scholarly publishing—one that could produce clean source code and encompass features essential for scholarly publishing, beyond the capabilities of typical 'web content' tools or the needs of Wikipedia.

Try 2: Substance.io

With the founding of Coko in 2015 a sophisticated, open-source editor tailored for scholarly collaboration still did not exist. So, I decided to invest in Substance.io, although it was also an immature solution, it was the only open-source framework available at the time for building editors. A significant portion of the investment I had received from the Shuttleworth Foundation for my fellowship—$150,000—was allocated to develop this framework. Substance.io held immense promise, designed to handle HTML, Coko's preferred format, as well as XML for those interested in building JATS editors. However, I was acutely aware of the limitations that sole reliance on our investment could impose on the project's longevity. Unlike the Wikimedia Foundation, which had no issue when it came to sustaining their internal technical developments, Substance.io consisted of just 2 people based in a small town in Austria struggling to make this work with no additional support.

To mitigate this, in 2016 I facilitated the formation of the "Substance consortium", bringing together organizations such as PKP and Érudit.

This collective effort aimed to build stakeholder support and (I hoped) eventually finance the ongoing development in the editor framework. PKP and Érudit had different goals to us, they needed a JATS editor, but our aim was not to force HTML on our collaborators, but to build a diverse set of invested stakeholders around the editor framework.

Despite our intentions, the founders of Substance pursued a direction that increasingly embedded JATS logic into the framework, making it more costly and complex for us to adapt to our needs. While the other members of the Texture Consortium were happy with this and were encouraging the development to go in this direction, it became evident that Substance was diverging from our requirements.

Try 3: ProseMirror

We felt very stuck. We had invested a lot in Substance.io only to see the project move away from our requirements. We felt very frustrated and felt there was a need for a critical review of our strategy.

The turning point came during a critical 2018 meeting in Athens with the Coko development team. After four days of intense deliberation, we concluded that continuing with Substance was untenable. The prospect of starting from scratch was daunting, given the resources and time already invested. However, the emergence of the recently open-sourced ProseMirror libraries, among a few other options, provided a glimmer of hope.

We made the difficult decision to begin anew. This decision birthed Wax, an editor that finally aligned with our vision for scholarly publishing. Wax represented not just a technological achievement but a testament to our resilience and commitment to the open-source ethos.

As it happens, in 2019 or so the Substance consortium died, but (thankfully) we had a new editor that proved to us we had made the right decision.

Reflections

This journey, while marked by setbacks, has been a profound learning experience. It underscored some vital lessons:

• Innovation Requires Resilience: Tackling innovation in the absence of foundational elements requires not just perseverance and flexibility, but also the fortitude for the long haul.
• Collaboration Needs Alignment: From the Substance process, we learned that for multi-stakeholder collaboration to endure and succeed, it's critical to establish clear, mutual objectives from the outset.
• Act Early on Change: For each of these unsuccessful changes, no matter how frustrating, we had to decide to bow out. Making changes proactively rather than reactively can save the day.
• Timing and Choice: Finding the Right Foundation: Sometimes, success hinges on the right starting point. It was the third foundation, ProseMirror, that proved to be the key. This experience taught us that with time, the ecosystem can sometimes evolve to reveal the optimal starting point.

© Adam Hyde, 2024, CC-BY-SA
Image public domain, created by MidJourney from prompts by Adam.