Markdown Hierarchy Parser

Parse Markdown into heading-addressable sections while preserving each section’s source text

from fastcore.test import *

Markdown documents already divide themselves into useful semantic units. HeadingDict preserves that hierarchy, so a long document can be inspected by section instead of searched and sliced as one flat string. Every node is a dictionary of numbered child sections with the complete Markdown for that section in .text.

Two kinds of long string dominate a big document’s cost, and both get replaced by numbers. Section addresses become dotted numbers - '1.4.5' for the fifth child of the fourth child of the first top-level heading - so an outline row is short to display and unambiguous to type back. And inline links - most of the bulk of an llms.txt index - are numbered in reading order: a URL costs 15-20 tokens to display, its only information content is which page it names, and quoting one invites a mistype, so sections render each [text](url) as [text][n], lynx-style. Link and Sections are the row types the listings below return; links, follow, and search act on the numbers directly.

---

source

Sections

def Sections(
    items:NoneType=None, counts:NoneType=None, **kw
):

Sections listed as addr title rows; each row is the live node

---

source

Link

def Link(
    n, # 1-based number, in document reading order
    txt, # The link text
    url, # The link target (never shown by `__repr__`; the number stands for it)
    tail, # The rest of the link's line, as context - an llms.txt entry's description
    line, # Document-absolute 1-based line number
):

One inline link, numbered in document reading order

Heading trees

HeadingDict supports normal dictionary lookup, keyed by 1-based section number. at('1.2.1') follows a dotted address in one call, paths() shows the outline - one addr title row per section, each row the live node - and find() returns a section when its title occurs exactly once anywhere. Addresses stay short to display and unambiguous to type back, however wordy or repetitive the headings are.

---

source

HeadingDict

def HeadingDict(
    src:str='', start_line:int=1, *args, **kwargs
):

A dictionary of numbered child sections, with the section Markdown in text.

---

source

create_heading_dict

def create_heading_dict(
    text, rm_fenced:bool=True, base:NoneType=None
):

Create a nested HeadingDict from Markdown headings, numbering sections and inline links in reading order.

sample_md = r'''# Hooks

Shared hook behavior.

## Common input fields

Every hook receives `session_id`.

## Hooks

### SessionStart

Runs at thread start.

```python
# This is code, not a heading
```

### PostCompact

Runs after compaction.'''

result = create_heading_dict(sample_md)
result.paths()

[('Hooks',),
 ('Hooks', 'Common input fields'),
 ('Hooks', 'Hooks'),
 ('Hooks', 'Hooks', 'SessionStart'),
 ('Hooks', 'Hooks', 'PostCompact')]

A node displays as its own addr title [size] line followed by up to two heading levels below it, so a bare display is the orientation step: the root shows the document’s top-level map, at() results show their neighborhood, and a leaf is a single line.

t = create_heading_dict('# A\n## B\n### C\n#### D')
test_eq(repr(t), '. [21]\n1 A [21]\n1.1 B [17]')
test_eq(repr(t.at('1.1')), '1.1 B [17]\n1.1.1 C [12]\n1.1.1.1 D [6]')
test_eq(repr(t.at('1.1.1.1')), '1.1.1.1 D [6]')
t

Use at() when you know the address, from an earlier paths() or search(). The returned node includes its heading and all Markdown beneath it, up to the next heading at the same or a higher level.

session = result.at('1.2.1')
session.text

'### SessionStart\n\nRuns at thread start.\n\n```python\n# This is code, not a heading\n```'

For non-left nodes, it shows the whole section including lower level:

hooks = result.at('1.2')
hooks.text

'## Hooks\n\n### SessionStart\n\nRuns at thread start.\n\n```python\n# This is code, not a heading\n```\n\n### PostCompact\n\nRuns after compaction.'

find() is shorter when a title is unique anywhere in the document. It raises when the title is absent or ambiguous, so it cannot quietly select the wrong section.

test_eq(result.find('SessionStart'), session)
test_fail(lambda: result.find('Hooks'), contains='found 2')

Fenced code and duplicate headings

Markdown examples often contain lines that look like headings. Backtick and tilde fenced blocks are ignored by default, so # This is code does not appear in the heading tree.

expected = r'''### SessionStart

Runs at thread start.

```python
# This is code, not a heading
```'''
test_eq(result.find('SessionStart').text, expected)
test_eq([n for n in result.paths() if n.title == 'This is code'], [])

Duplicate siblings

Real documents repeat sibling titles freely - changelog entries, or API pages with a Parameters heading per function. Numeric keys make that a non-event: each sibling gets its own number, so every section stays addressable. find() still refuses ambiguity, naming the candidate addresses, so a title lookup cannot quietly select the wrong section.

dup = create_heading_dict('# A\n## Same\n## Same')
test_eq([f'{n.addr} {n.title}' for n in dup.paths()], ['1 A', '1.1 Same', '1.2 Same'])
test_fail(lambda: dup.find('Same'), contains='found 2')

Searching sections

Questions about a big document usually arrive as “where does it talk about X”, not “what is its shape” - so search() is the entry point when a document is too long to read whole. It matches a case-insensitive regex line by line (an invalid regex matches literally), and attributes each hit to the deepest section containing its line. That last part matters: every ancestor’s text contains every descendant’s, so matching whole sections instead of lines would always return the root. Results come back as addr title (count) rows in document order, and the address is what you type next:

---

source

HeadingDict.search

def search(
    pat, # Case-insensitive regex, matched line by line; an invalid regex matches literally
):

The deepest sections owning a line matching pat, in document order, with match counts

hits = result.search('runs')
test_eq([f'{n.addr} {n.title}' for n in hits], ['1.2.1 SessionStart', '1.2.2 PostCompact'])
test_eq(hits[0], result.at('1.2.1'))
hits

Every listing row ends with the section’s size: a humanized count of its source characters, subsections included. The whole-or-sections decision (see read_md’s ~30k guidance) then reads straight off any listing, with no separate length check.

sized = create_heading_dict('# A\n## Same\n## Same')
test_eq(repr(sized.paths()), '1 A [19]\n1.1 Same [7]\n1.2 Same [7]')
sized.paths()

Line-addressed views

Each node records start_line, the 1-based line number of its heading in the original document, and view() can prefix each line of the section with its absolute line number or with a lineno|hash| address. The address comes from fastcore.tools.lnhash (the same CRC-32 formula exhash uses), so when the parsed text came from a local file, the output of view(lnhashs=True) provides valid exhash addresses for editing that file directly.

result.find('PostCompact').view(lnhashs=True)

nums=True shows plain line numbers instead; lnhashs wins when both are set, and with neither, view() is the source exactly as stored in src. That is the same as text until a document contains links: text renders them numbered, and an edit address is only good against the bytes really in the file, so view() never does. A section starts at its own heading line, so numbering lines up with the full document.

sec = result.find('PostCompact')
test_eq(sample_md.splitlines()[sec.start_line-1], '### PostCompact')
test_eq(sec.view(lnhashs=True).splitlines()[0], lnhash(sec.start_line, '### PostCompact')+'### PostCompact')
test_eq(sec.view(nums=True).splitlines()[0], f"{sec.start_line}: ### PostCompact")
test_eq(sec.view(nums=True, lnhashs=True), sec.view(lnhashs=True))
test_eq(sec.view(), sec.src)

Files

---

source

create_heading_dict_file

def create_heading_dict_file(
    path, rm_fenced:bool=True
):

Create a HeadingDict from the Markdown file at path (~ ok), recording path for refresh and as the base for follow

create_heading_dict_file reads a Markdown file (expanding a leading ~) and records the path on the returned root, so the whole local-file workflow needs no raw file reads. Since a parsed tree persists while the file may change, addresses can go stale after an edit: refresh() re-reads the file and returns a fresh tree. A stale address fails loudly at exhash’s hash check rather than editing the wrong line.

p = Path('tmp_sample.md')
p.write_text(sample_md)
d = create_heading_dict_file('tmp_sample.md')
test_eq(d.find('PostCompact').view(lnhashs=True), result.find('PostCompact').view(lnhashs=True))
test_eq(d.refresh().paths(), d.paths())
p.unlink()

Links

The workflow this module exists for starts with a docs site’s llms.txt: fetch the table of contents, pick the one page the task needs, fetch that. Here is Claude Code’s:

toc_txt = httpx.get('https://code.claude.com/docs/llms.txt').text
len(toc_txt)

Far too long to display for a task that typically needs one entry - and most of the bulk is URLs, which are the worst text an LLM can spend context on: each costs 15-20 tokens, its only payload is which page it names, and quoting one invites a mistype. links() lists what the parse numbered; a pattern filters over each link’s text, target, and context, so the listing stays as small as the question:

---

source

HeadingDict.links

def links(
    pat:str='', # Case-insensitive regex matched against each link's text, target, and tail
):

This section’s Link rows, numbered document-wide, filtered by pat

---

source

Links

def Links(
    items:NoneType=None, *rest, use_list:bool=False, match:NoneType=None
):

A list of Link rows, displayed one per line

toc = create_heading_dict(toc_txt)
assert len(toc.links()) > 100
toc.links('subagent')

A row shows its number, text, and the rest of its line - for an llms.txt entry, exactly the page description. The URL is carried on the row but never displayed; the number stands for it. Sections render the same way: text shows every parsed link as [text][n], so displaying any part of a link-heavy document stops paying the URL tax, while view() and src keep the file’s real bytes for editing:

assert toc.src == toc_txt
print(toc.text[toc.text.find('- ['):][:370])
len(toc.text)/len(toc_txt)

The precise contract, on something small: numbering is reading order; fenced lines and images stay out of the table, matching the heading scan; tail is the link’s line after the link itself, any leading colon stripped, so llms.txt descriptions come through clean; and a node’s links() sees only its own line range.

links_md = '''# Guide

Start with the [install](install.md): five minutes.

## Reference

- [API](https://example.com/api.md): every function
- ![logo](logo.png) and a [changelog](notes.md)

```md
[fenced](ignored.md)
```'''

g = create_heading_dict(links_md)
test_eq([(l.n, l.txt, l.tail) for l in g.links()],
        [(1, 'install', 'five minutes.'), (2, 'API', 'every function'), (3, 'changelog', '')])
test_eq([l.txt for l in g.at('1.1').links()], ['API','changelog'])
g.links('example.com')

Rendered, the section reads with numbers in place; the image, the fenced line, and the stored source are all untouched:

ref = g.at('1.1')
assert '[API][2]' in ref.text and '![logo](logo.png)' in ref.text and '[fenced](ignored.md)' in ref.text
test_eq(ref.view(), ref.src)
print(ref.text)

Following links

follow completes the loop: a number from any listing in, the target’s text out, no URL touched in between. It deliberately returns a plain str - parse it with create_heading_dict when the tree is wanted, and skip parsing when it is not:

---

source

HeadingDict.follow

def follow(
    n, # A link number, as shown by `links`
):

The target text of link n: fetched for http(s) targets, read from disk for local ones (resolved against base)

l = toc.links('Create custom subagents')[0]
page = create_heading_dict(toc.follow(l.n))
assert 'subagent' in page.src.lower()
page.paths(2)

Whether to read a fetched page whole is a length question: under roughly 30k characters, display text in full - the outline-and-sections dance would cost more than it saves, and the numbered links have already cheapened the whole thing. This page is bigger than that, so search() picks the sections worth pulling:

assert len(page.src) > 30_000
hits = page.search('frontmatter')
assert hits
hits

The whole llms.txt routine is now four short steps with no URL in any of them: parse the index, filter the links, follow by number, parse the page. Relative targets resolve against base, the document’s own location: create_heading_dict_file records it automatically, and for parsed strings pass base= - a URL works too, via urljoin. Without one, a relative link fails loudly rather than guessing, and so does a number outside the table:

p,inst = Path('tmp_index.md'),Path('install.md')
p.write_text(links_md)
inst.write_text('# Install\n\nRun the thing.')
fdoc = create_heading_dict_file('tmp_index.md')
test_eq(fdoc.follow(1), '# Install\n\nRun the thing.')
test_fail(lambda: g.follow(1), contains='base')
test_fail(lambda: g.follow(99), contains='1..3')
p.unlink(); inst.unlink()