Thwarted for the last time!

I can always tell when I haven't written a blog in a while, because inevitably on running gatsby develop, I will get a pile of inscrutable errors (despite not touching anything for months).

⠋ compile gatsby files
node(52176,0x1709bf000) malloc: Incorrect checksum for freed object 0x12110aa00: probably modified after being freed.
Corrupt value: 0x0
node(52176,0x170dcb000) malloc: Incorrect checksum for freed object 0x1221f8400: probably modified after being freed.
Corrupt value: 0x0
node(52176,0x1709bf000) malloc: *** set a breakpoint in malloc_error_break to debug
⠦ compile gatsby files

When you're getting memory allocation errors on npm start, it's time to step away from the computer

Well that ends today (I hope).

Part of my issue was that I was using a custom mdx fork of gatsby-starter-lumen, which is a theme that doesn't seem to be maintained anymore. What's worse, mdx support in Gatsby also seems to be.... less than great. The current solution to the inevitable npm i dependency hell is to pass --legacy-peer-deps when using gatsby-plugin-mdx, which seems broken for an officially supported mode (that you toggle on when you start a new Gatsby project!)

I'd chosen Gatsby/MDX initially because I work with React on a daily basis, and I wanted to include Observable Notebook components in blog posts (and there was a nice little export function you could run to achieve this). I did this in a few places for some interactive visualizations, which suited my blogging style. However, the Gatsby headache kept mounting, introducing additional friction to something that I should enjoy doing, and should be much more seamless. (I spend enough time debugging React for $$; it's not something I want to do in my free time).

The Approach

I started with the recommended create script:

npx @observablehq/framework@latest create

Markdown Files

Since many of my blog posts were already in boring markdown, I just dumped the directories from gatsby into my src directory to see what would happen.

/Users/mclare/
  ── workspaces/
      └── mclare-blog/
         ├── dist/
         └── posts/
            └── 2020-09-18---Building-Blender-As-A-Python-module

Posts with .md extensions were immediately navigable and viewable at /posts/2020-09-18---Building-Blender-As-A-Python-module. I needed to adjust the paths on the image files, but that was the only main change needed to the files.

However, Gatsby had previously processed these directories to automatically appear at a nicer slug like /posts/building-blender-as-a-python-module, so I ended up just bulk renaming the directories to match their slugs in the frontmatter rather than finding a similar workaround.

MDX Files

The posts that were in .mdx had the aforementioned Observable/React components. Importing and calling them in a mdx file looked something like this:

import {
  SoftwareEngineerInterviews,
  WeekByWeekInterviews,
  GroupedInterviewType,
} from "./observable/softwareInterview.js";

<SoftwareEngineerInterviews />

Observable has a helpful guide for converting notebooks, so that's what I used as a reference here. I can definitely go back and clean up the code at this point (possibly make these into components), but it worked as is with a few tweaks. I removed the small amount of React I was using to just have pure D3 in the JS cells.

Sorting Pages

It's a bit confusing to figure out where to troubleshoot things with Observable Framework, as it appears most folks have moved to GitHub discussions rather than the previous iteration on Observable Talk. However, it turns out that there's an enthusiastic group in the Observable community already tackling the problem of blogging with Observable Framework. I was able to adapt a load-posts.js script from James King to automatically update my sidebar pages within my home page and at /posts in descending date order thanks to the existing date field on the front matter:

import fs from "fs";
import path from "path";
import matter from "gray-matter";
import { fileURLToPath } from "url";
import { dirname } from "path";

// Get the current directory (since __dirname is not available in ES modules)
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

// Define the posts directory path
const postsDirectory = path.join(__dirname, "..", "src", "posts");

// Function to load markdown files from the posts directory
function loadMarkdownFiles() {
  const postFolders = fs
    .readdirSync(postsDirectory, { withFileTypes: true })
    .filter((dirent) => dirent.isDirectory())
    .map((dirent) => dirent.name);

  const postsData = postFolders
    .map((folder) => {
      const fullPath = path.join(postsDirectory, folder, "index.md");

      try {
        const fileContent = fs.readFileSync(fullPath, "utf8");

        // Parse front matter using gray-matter
        const { data: frontMatter, content } = matter(fileContent);

        return {
          slug: folder,
          frontMatter,
          fileName: folder,
          content,
        };
      } catch (error) {
        console.warn(
          `Warning: Could not read file for post "${folder}". Skipping this post.`
        );
        return null;
      }
    })
    .filter((post) => post !== null)
    .filter((post) => post?.frontMatter?.draft !== true)
    .filter((post) => post?.frontMatter?.title !== undefined);

  const outputData = postsData.sort((a, b) => {
    const dateA = new Date(a.frontMatter.date);
    const dateB = new Date(b.frontMatter.date);
    return dateB - dateA; // Sort in descending order (newest first)
  });

  return outputData;
}

// Load posts and export as default
const posts = loadMarkdownFiles();
export default posts;

CSS

The lumen theme used CSS modules and Sass. I think it's likely that Observable will at some point support CSS modules (it might already, tbh), but I'll doubt they'll go in on all the different CSS management frameworks. I ended up using a lot of sass ${lumen-dir} ${framework-dir} to extract all the CSS I needed to get the same feel in my post feed and posts themselves, and just dumped the results into a style.css at the root of the project as per the Observable instructions. I also did some not so great things with the page loader to get the right DOM structure for a custom sidebar on the home page that I'd like to clean up later (there are definitely some layout issues).

Before (Gatsby)

Gatsby Home Page

After (Observable Framework)

Observable Framework Home Page

RSS Feed

Turns out in addition to solving the page listing problem, James King has already also tackled the RSS feed issue. I had a few minor modifications due to differences in how we structured our directories, but this was pretty straightforward to add and then run post build.

Future Work

Things that are currently missing:

Things I need a better way of handling:

Things I'll probably omit:

New features I'm liking:

What I'm most excited about with this approach is that I'll be able to update some of my visualizations automatically via data loaders to maintain current data. It would be particularly interesting to have a live view on my Los Angeles' Soft Story Retrofits and the previous work I did on tracking structural engineering licensure.

Onwards and upwards!