C-xrefactory - a C/Yacc refactoring and code browsing tool - Design Documentation

A user can navigate all references of a symbol, limited to the semantic scope of that symbol, by "Goto Definition" and then navigate using "Next/previous Reference". This is a fast way to inspect where a symbol is used.

A user can search for symbols by name using two operations:

Both operations accept wildcard patterns: matches any sequence of characters and ? matches a single character. For example, parse finds all symbols starting with "parse", while get_? finds three-letter symbols starting with "get_".

The search results are presented in a list. Use the up/down arrow keys to move between entries. Press RET to inspect a symbol — this navigates to its definition and enters the reference browser where Next/Previous Reference can be used to visit all references.

Pressing p and n in the search results navigates the search history — returning to the results of a previous or next search, not moving between entries in the current list.

As c-xrefactory have information about symbols and their semantic scope, it can also provide semantically informed completions and suggestions.

In his book "Refactoring" Martin Fowler describes a large number of refactorings, changes to source code that does not change the behaviour, but improves it structure and readability. For each refactoring they describe step by step which edits to make to apply the refactoring manually.

The natural next step was of course to attempt to automate this in editors or IDEs, which started to happen.

In an article from 2001 Martin pronounces Xref, the ancestor to c-xrefactory, to be the first tool to cross "Refactorings Rubicon", being able to extract a function semantically correct.

The term "automated" means that some software can examine the source code and quickly and safely modify it using patterns from the list of possible refactorings, without user interaction. Many refactorings in the book, and on the website, are applicable mostly for OO-languages, but many also apply to C. c-xrefactory can perform some of them. More are considered for implementation.

Using these automated refactorings it is much easier and safer to continuously maintain and improve the quality of any code base.

The LSP protocol was designed for single-shot, non-interactive operations. This creates constraints for c-xrefactory’s advanced refactorings:

Interactive Refactorings: C-xrefactory’s extract/parameter operations require multi-step user input (names, positions, declarations). LSP’s textDocument/codeAction doesn’t support interactive dialogs.

Symbol Browsing: C-xrefactory provides interactive symbol browsers with filtering and keyboard navigation. LSP returns flat reference lists with no standard for interactive UI.

Strategy: The LSP implementation aims to:

LSP serves to make c-xrefactory more accessible while the Emacs client probably will remain the gateway to its complete refactoring power.

cxrefProgram is the core container. It does all the work when it comes to finding and reporting references to symbols, communicating refactoring requests as well as storing reference information for longer term storage and caching.

Although c-xref can be used as a command line tool, which can be handy when debugging or exploring, it is normally used in "server" mode. In server mode the communication between the editor extension and the 'cxrefProgram` container is a back-and-forth communication using a non-standard protocol over standard pipes.

The responsibilities of cxrefProgram can largely be divided into

At this point it seems like refactorings are performed as separate invocations of c-xref rather than through the server interface.

The EditorExtension container is responsible for plugging into an editor of choice and handle the user interface, buffer management and executing the refactoring edit operations.

Currently there is only one such extension supported, for Emacs, although there existed code, still available in the repo history, for an extension for jEdit which hasn’t been updated, modified or checked for a long time and no longer is a part of this project.

The References database stores crossreferencing information for symbols visible outside the module it is defined in. Information about local/static symbols are not stored but gathered by parsing that particular source file on demand.

Currently this information is stored in a somewhat cryptic, optimized text format.

This storage can be divided into multiple files, probably for faster access. Symbols are then hashed to know which of the "database" files it is stored in. As all crossreferencing information for a symbol is stored in the same "record", this allows reading only a single file when a symbol is looked up.

Parse C and Yacc source files, producing symbol references and semantic information for the reference database and for feature-specific operations (completion, extraction, refactoring).

The parsing component consists of several internal modules:

The integrated preprocessor is a key architectural choice: by implementing its own preprocessor rather than using the system’s, c-xrefactory can navigate to macro definitions, show macro usage, refactor macro names, and complete macro identifiers. The trade-off is imperfect compatibility with all compiler-specific preprocessor extensions.

The parser’s behavior is configured through ParserOperation, decoupling parsing from server-level concerns:

The server maps its ServerOperation to a ParserOperation via getParserOperation(), so the parser never needs to know about server-level operation enums.

The parsers are generated using a patched Berkeley yacc (byacc-1.9). The patch modifies the skeleton to support error recovery and a recursive parsing feature (originally for Java). CPP macros rename parser data structures so that multiple parsers can coexist in the same executable. The Makefile generates and renames the parser output files.

Key entry points (see parsing.h):

parseToCreateReferences() is the clean entry point used by LSP mode, the entry-point reparse loop, and navigation refresh. It takes a filename, determines the language, sets up configuration, and parses.

callParser() is the lower-level dispatch used by server mode’s parseInputFile(), where configuration is set up separately with cursor position and operation-specific state.

Build and update the on-disk cross-reference database (.cx files) by parsing all scheduled project files. The xref component is the batch counterpart to the interactive server: where the server processes one file per editor request, xref processes all project files in a single invocation.

When the in-memory reference table overflows mid-parse, xref flushes the accumulated references to disk via saveReferences(), recovers memory, and continues parsing the remaining files. This setjmp/longjmp-based overflow mechanism allows xref to handle projects larger than available memory.

callXref() is also called by the refactory to re-index files after refactoring operations.

Serve the editor extension by processing requests in an infinite loop. Each request is a set of command-line-style options received over stdin. The server:

The server runs as a long-lived process started by the editor (c-xref -server). Communication is over stdin/stdout using a text protocol where commands look like command-line options.

Before dispatching any operation, the server reparses stale preloaded files so that all operations see fresh in-memory references. This separates the concern of "keeping data fresh" from individual operation logic.

The entry-point reparse loop:

The needsBrowsingStackRefresh flag bridges the entry-point (which handles parsing) and the navigation module (which handles browsing stack updates). Navigation checks this flag instead of fileNumberIsStale() and calls updateSessionReferencesForFile() to update only the browsing stack without re-parsing.

After a browsing stack refresh, the server restores its position by finding the nearest reference to where it was, then advances (NEXT) or retreats (PREVIOUS) in list order. This respects the definition-first navigation ordering without assuming any particular sort order.

The server can only reparse and navigate files it knows about. "Knowing" a file means it has an entry in the file table — a file number and metadata (FileItem). But being known does not mean being parsed: a file in the table may have no references in the referenceableItemTable yet. This distinction matters because the entry-point reparse loop, the reverse-include graph walk, and navigation all operate only on known files.

Files enter the file table through three mechanisms:

Disk database (loadFileNumbersFromStore()). On the first request that triggers project initialization, the server loads file numbers from the .cx database. This is why -create was historically required before any interactive use: it populated the persistent database that seeds the file table. Without it, the server would only know about the single file in the current request.

Editor preloads. When the editor sends -preload <file> <tmpfile>, the file enters the file table via the editor buffer mechanism. Files created during a session — new source files the user opens — are discovered this way without any explicit rescan.

Project directory glob (processFileArguments()). At startup, after project context initialization, the server walks options.inputFiles (typically ., the project root) recursively, filtering by known source suffixes (.c, .y, etc.) and honoring -prune paths. This discovers all compilation units in the project directory without requiring a prior -create invocation.

The three mechanisms are complementary: the disk database provides historical knowledge, preloads provide real-time updates during the session, and the project directory glob provides a fresh scan of what actually exists on disk at startup.

Design boundary: Files that appear on disk but are not in the file table — because they weren’t in the disk database, weren’t opened in the editor, and weren’t present at startup — are invisible to the server until the next session. The startup glob combined with editor preloads covers the primary use cases: all project files are discovered at startup, and new files created during the session enter through the editor.

Operations are classified by what they need:

Implement the Language Server Protocol interface, allowing LSP-capable editors (VS Code, Emacs lsp-mode, etc.) to use c-xrefactory’s parsing and navigation capabilities. Currently at proof-of-concept stage with partial textDocument/definition support.

The LSP adapter is a self-contained subsystem with clear internal layering:

The LSP adapter takes a fundamentally different approach from the legacy editor server:

The LSP adapter is a proof-of-concept. Key gaps:

Coordinate refactoring operations: rename, move, extract, and argument manipulation. The refactory component receives a refactoring request, uses the parser and reference database to analyze the code, performs safety checks, and produces a sequence of edits for the editor to apply.

Refactoring operations run as a separate c-xref invocation, not through the long-lived editor server. The editor starts a new c-xref process with -refactory and the specific refactoring flag (e.g. -rfct-rename, -rfct-extract-function). This separate process communicates results back via the protocol and exits when done.

This design means the refactoring process has its own option state (refactoringOptions), separate from the server’s. The check options.refactoringRegime == RegimeRefactory gates refactoring-specific code paths.

A typical refactoring (e.g. rename):

Before applying a refactoring, the refactory performs safety checks (OP_INTERNAL_SAFETY_CHECK). These use the reference database to verify that the transformation is semantically valid — for example, that a rename won’t collide with an existing symbol in scope.

Analyze control flow for extract-function/macro/variable refactoring. The extract component operates in two phases: collection during parsing (registers synthetic labels, gathers references) and analysis after parsing (classifies variables, generates output).

Extraction uses a specialized parse operation (PARSE_TO_EXTRACT) that tracks:

After parsing, it classifies variables as inputs (passed as parameters), outputs (returned), or local (moved into the extracted function), and generates the function signature, call site, and body.

Manage the in-memory symbol reference tables and the browsing session stack. The cxref component is the runtime engine for symbol lookup: it loads references from the on-disk database (via cxfile), merges them with freshly parsed references, and provides the data structures that navigation and refactoring operations query.

c-xrefactory’s core functionality relies on a symbol database that stores cross-references, definitions, and usage information for all symbols in a project. The database has two forms:

Browsing Stack (sessionData.browsingStack)

The browsing stack is the runtime data structure for symbol navigation. Each push operation (triggered by a navigation request) creates a new session entry containing the references for the symbol under the cursor.

Referenceable Items

A ReferenceableItem represents a symbol (function, variable, macro, type, etc.) with its attributes: link name, type, storage class, scope, and visibility. Each referenceable item has a linked list of Reference entries recording every usage position.

References

A Reference records a single occurrence of a symbol: file, line, column, and usage kind (defined, declared, used, etc.). References are the fundamental unit that navigation, unused-symbol detection, and refactoring operate on.

When the user requests "go to definition" for a symbol:

The Memory module provides arena-based allocation for performance-critical and request-scoped operations:

Historical Context

In the 1990s when c-xrefactory originated, memory was scarce. The design had to:

Most memory arenas use statically allocated areas. Only cxMemory supports dynamic resizing to handle out-of-memory situations by discarding, flushing and reusing memory. This forced implementation of a complex caching strategy since overflow could happen mid-file.

Modern Benefits

Even with abundant modern memory, arena allocators provide:

Marker-Based Cleanup

Functions save a marker before temporary allocations:

Buffer Growth Pattern

Long-lived buffers that may need to grow:

Overflow Handling

The cxMemory arena supports dynamic resizing:

When overflow occurs, handler can:

Key functions (see memory.h):

See the "Arena Allocator Lifetime Violations" section in the Development Environment chapter for:

Modern systems have abundant virtual memory. Possible improvements:

The experimental FlushableMemory type explores some of these ideas but hasn’t replaced current implementation.

Content buffers provide an in-memory cache of file content that transparently overrides disk file reading during parsing. When a content buffer exists for a file, the parser uses its in-memory content instead of reading from disk.

Content buffers (the EditorBuffer struct) serve three distinct roles:

The lexer reads from a character buffer (currentFile.characterBuffer), which is the common layer regardless of content source. Where the character buffer gets its data depends on whether a content buffer exists:

When the lexer processes an #include, it calls findOrCreateAndLoadEditorBufferForFile() for the included file. This function:

If a content buffer is found, the character buffer is set up to read from it. If not (file doesn’t exist or is a directory), the fallback openFile() path reads from disk via a FILE handle. This means preloaded editor content transparently overrides disk content — the parser doesn’t know or care where the content came from.

Buffer lifetime depends on the server mode and buffer role:

The existing bufferIsCloseable() predicate distinguishes these cases: it returns false for preloaded buffers (preserving editor content) and true for loaded, unmodified, marker-free buffers (allowing cache cleanup).

For preloaded buffers, staleness is detected by comparing fileItem→lastParsedMtime with buffer→modificationTime. After parsing a preloaded buffer, lastParsedMtime is set to the buffer’s modification time. If the buffer is re-sent with a new tmp file (new mtime), the file appears stale and is reparsed.

See the Server component’s Entry-Point Reparse section for how staleness drives the sync phase.

Read and write the CXref database (.cx files) in a compact text format. Cxfile is the persistence layer: it serializes symbol references to disk after parsing and loads them back during navigation and refactoring operations.

The database uses a hash-partitioned file structure:

All information about a symbol is stored in exactly one file, determined by hashing its link name. This means a single file read suffices to look up any symbol.

Records use the general format <number><key>[<value>]. The encoding uses single-character markers listed at the top of cxfile.c.

The coding often starts with a number followed by a character key: 4l means line 4, 23c means column 23. References are optimized to avoid repeating fields that haven’t changed — so 15l3cr7cr means two references on line 15, one at column 3, the other at column 7 (using fsulc fields: file, symbol index, usage, line, column).

Some fields carry a length prefix: filenames use <length>:<path> (e.g. 84:/home/…​/file.c), version strings use <length>v.

Example file information line:

Reading is controlled by scanFunctionTable arrays. Each entry maps a record key to a handler function. As the reader encounters a key in the file, it looks up the handler and calls it. This table-driven approach allows different consumers to register for different record types — for example, loading only symbol names vs. loading full reference lists.

The Emacs client extension provides the user-facing interface for navigation, refactoring, and completion. It starts the c-xref process on the first user interaction and communicates with the server process over stdin/stdout using a text protocol.

To give the server access to unsaved content the client sends -preload <file> <tmpfile> for modified editor. On PUSH, NEXT, and PREVIOUS, all modified buffers (including the current one if modified) are preloaded. Unmodified buffers are not preloaded — the server reads the disk file directly.

The preload mechanism writes the buffer content to a temporary file and passes both the original filename and the temp file path. The server creates an EditorBuffer with the content and the temp file’s modification time. After each request, all editor buffers are closed (closeAllEditorBuffers), so preloads are re-sent on every request. == Implementation Notes

This chapter collects cross-cutting implementation details that don’t belong to a single component: editor-server protocol, file processing orchestration, multi-pass configuration, and other observations about how the subsystems interact.

All modes begin by marking files for processing using the isScheduled flag in the file table.

Initial Scheduling (All Modes)

Called from:

Flow:

Result: All command-line files (and directory contents if -r flag) are marked with isScheduled = true.

[TARGET] processFileArguments() will be replaced by scanProjectForFilesAndIncludes(), which discovers CUs by globbing the project directory rather than relying on command-line file lists. This also builds the include graph transitively by text-scanning #include lines and resolving paths via options.includeDirs. See Target: Unified Server Flow (ADR 22).

Additional Update Scheduling (Xref Mode Only)

Called from: callXref() → scheduleModifiedFilesToUpdate() [xref.c:296]

Flow:

Result: In update mode, only modified files (and their includers) get isScheduled = true.

SINGLE FILE PER REQUEST - Server processes one file per request.

The entry refresh (see Terminology in Principles) ensures the reference table is up-to-date before each operation. Pass ordering is critical: Pass 1 reparses stale CUs, which updates their TypeCppInclude references. Pass 2 then queries those freshened references to find which CUs include stale headers. Pass 3 uses the include structure from the lightweight scan to find CUs sharing headers with the request file that have never been parsed. A user cannot create a new include edge without editing the includer, so the includer is always stale when the edge is new.

The per-request dispatch still follows the legacy pattern:

PROCESSES ALL SCHEDULED FILES - Xref creates a list of all scheduled files and processes them in a loop.

Assignments to `inputFileName`

Server Mode: Set once per file processed

Xref Mode: Set ONCE per file

`requestFileNumber` Only Used in Server Mode

Confusion: THREE Different File Number Globals!

In xref.c line 160:

But currentFileNumber is defined in parsing.c:26:

The comment reveals the distinction: currentFileNumber can change DURING parsing when entering #include files, while requestFileNumber stays constant as "the file we were asked to process."

Double initInput() Call in Xref Mode

In Xref mode, initInput() is called TWICE for the same file:

This appears to be a bug or wasteful duplication.

`initializeFileProcessing` is Heavy Orchestration

This 500-line function does five major phases:

The firstPass parameter gates Phase 4’s memory checkpoint save/restore.

[TARGET] initializeProjectContext will be eliminated — it duplicates phases 1-4 of initializeFileProcessing. With the unified flow, all initialization goes through loadProjectSettings() once at startup. The multi-project fast-path optimization (checkpoint restore when same project) becomes dead code with single-project servers (ADR 21).

Naming Inconsistency

If they represent the same concept (the file being processed), they should have parallel names.

Server Mode - Single File Per Request

Xref Mode - All Files Per Invocation

LSP Mode - Single File Per Request

The current architecture has two initialization paths (initializeProjectContext for OP_ACTIVE_PROJECT, initializeFileProcessing for legacy per-file) and relies on the disk database as the source of truth for project structure. The target design unifies these into a single path where the in-memory database is authoritative.

Key principles:

This design eliminates the need for -create (replaced by the scan) and for callXref() before refactoring (replaced by scan + incremental reparse of stale files). It also makes the server viable for LSP, where blocking on a full project parse at startup is not acceptable.

The Server and Xref paths do similar things but with different names and structures, creating cognitive overhead. Potential improvements:

The paths are intertwined but different, making it hard to keep in your head which one you’re modifying. Making them more similar where possible would reduce cognitive load during refactoring work.

Lexing/scanning is performed in two layers, one in lexer.c which seems to be doing the actual lexing into lexems which are put in a lexembuffer. This contains a sequence of encoded and compressed symbols which first has a LexemCode which is followed by extra data, like Position. These seems to always be added but not always necessary.

The higher level "scanning" is performed, as per ususal, by yylex.c. lexembuffer defines some functions to put and get lexems, chars (identifiers and file names?) as well as integers and positions.

At this point the put/get lexem functions take a pointer to a pointer to chars (which presumably is the lexem stream in the lexembuffer) which it also advances. This requires the caller to manage the LexemBuffer’s internal pointers outside and finally set them right when done.

It would be much better to call the "putLexem()"-functions with a lexemBuffer but there seems to be a few cases where the destination (often dd) is not a lexem stream inside a lexemBuffer. These might be related to macro handling.

As the refactoring functions need some amount of semantic information, in the sense of information gathered during parsing, this information is collected in various ways when c-xref calls the "sub-task" to do the parsing required.

Two structures hold information about various things, among which are the memory index at certain points of the parsing. Thus it is possible to verify e.g. that a editor region does not cover a break in block or function structure. This structure is, at the point of writing, called parsedInfo and definitely need to be tidied up.

The editor invokes a new c-xref process for the following cases:

There is some magical editor buffer management happening inside of c-xref which is not clear to me at this point. Basically it looks like the editor-side tries to keep the server in sync with which buffers are opened with what file…​

At this point I suspect that -preload <file1> <file2> means that the editor has saved a copy of <file1> in <file2> and requests the server to set up a "buffer" describing that file and use it instead of the <file1> that recides on disk.

This is essential when doing refactoring since the version of the file most likely only exists in the editor, so the editor has to tell the server the current content somehow, this is the -preload option.

The initial invocation of the edit server creates a process with which communication is over stdin/stdout using a protocol which from the editor is basically a version of the command line options.

When the editor has delivered all information to the server it sends 'end-of-option' as a command and the edit server processes whatever it has and responds with <sync> which means that the editor can fetch the result in the file it named as the output file using the '-o' option.

Setting the emacs variable c-xref-debug-mode forces the editor to copy the content of such an output file to a separate temporary file before re-using it.

For some interactions the editor starts a completely new and fresh c-xref process, see below. And actually you can’t do refactorings using the server, they have to be separate calls. (Yes?) I have yet to discover why this design choice was made.

Communication between the editor and the server is performed using text through standard input/output to/from c-xref. The protocol is defined in src/protocol.tc and must match editor/emacs/c-xrefprotocol.el.

The definition of the protocol only caters for the server→editor part, the editor→server part consists of command lines resembling the command line options and arguments, and actually is handled by the same code.

The file protocol.tc is included in protocol.h and protocol.c which generates definitions and declarations for the elements through using some macros.

There is a similar structure with c-xrefprotocol.elt which includes protocol.tc to wrap the PROTOCOL_ITEMs into defvars.

There is also some Makefile trickery that ensures that the C and elisp impementations are in sync.

One noteable detail of the protocol is that it carries strings in their native format, utf-8. This means that lengths need to indicate characters not bytes.

The editor fires up a server and keeps talking over the established channel (elisp function 'c-xref-start-server-process'). This probably puts extra demands on the memory management in the server, since it might need to handle multiple information sets and options (as read from a .cxrefrc-file) for multiple projects simultaneously over a longer period of time. (E.g. if the user enters the editor starting with one project and then continues to work on another then new project options need to be read, and new reference information be generated, read and cached.)

FINDINGS:

The editor server is started using the appropriate command line option and then it keeps the communication over stdin/stdout open.

The editor part sends command line options to the server, which looks something like (from the read_xrefs test case):

In this case the "-olcxpush" is the operative command which results in the following output

As we can see from this interaction, the server will handle (all?) input as a command line and manage the options as if it was a command line invocation.

This explains the intricate interactions between the main program and the option handling.

The reason behind this might be that a user of the editor might be editing files on multiple projects at once, so every interrogation/operation needs to clearly set the context of that operation, which is what a user would do with the command line options.

The refactoring protocol messages exchanged between the refactory process and the editor:

Rename Example

Invocation: -rfct-rename -renameto=NEW_NAME -olcursor=POSITION FILE

Result: A sequence of precheck/replacement pairs:

followed by:

Extraction Example

Extraction (-rfct-extract-function) returns an <extraction-dialog> with three <str> parts: the call site replacement, the text to insert before the region (function header), and the text to insert after (closing brace). The editor applies these and then initiates a rename for the new function name.

Protocol Messages

Memory allocation is managed through the Memory structure, which implements an arena/bump allocator. Different memory arenas serve different purposes:

See Components chapter for detailed description of each arena’s purpose and lifetime.

The optMemory arena requires special handling because Options structures are copied during operation. When copying, all pointers into option memory must be adjusted to point into the target structure’s memory area, not the source’s.

Functions like copyOptions() perform this pointer adjustment through careful memory arithmetic, traversing a linked list of all memory locations that need updating.

There are three possible sources for options.

Not all options are relevant in all cases.

All options sources uses exactly the same format so that the same code for decoding them can be used.

When the editor has a file open it needs to "belong" to a project. The logic for finding which is very intricate and complicated.

In this code there is also checks for things like if the file is already in the index, if the configuration file has changed since last time, indicating there are scenarios that are more complicated (the server, obviously).

But I also think this code should be simplified a lot.

A ReferenceableItem represents a referenceable entity in the codebase - something that can be referenced from multiple locations:

Each ReferenceableItem contains:

ReferenceableItems are stored in the referenceableItemTable (hash table) and persisted to .cx files.

A Reference represents a single occurrence of a ReferenceableItem at a specific location:

Each ReferenceableItem maintains a linked list of all its References, allowing you to find every place that entity appears in the codebase.

Note: The term "Reference" in this context means "occurrence" - one specific use of an entity at one location. This is distinct from C++ references or reference semantics.

Options has its own allocation using optAlloc which allocates in a separate area, currently part of the options structure and utilizing "dynamic allocation" (dm_ functions on the Memory structure).

The Options structure are copied multiple times during a session, both as a backup (savedOptions) and into a separate options structure used by the Refactorer (refactoringOptions).

Since the options memory is then also copied, all pointers into the options memory need to be updated. To be able to do this, the options structure contains lists of addresses that needs to by "shifted".

When an option with a string or string list value is modified the option is registered in either the list of string valued options or the list of string list valued options. When an options structure is copied it must be performed using a deep copy function which "shifts" those options and their values (areas in the options memory) in the copy so that they point into the memory area of the copy, not the original.

After the deep copy the following point into the option memory of the copy

Arena allocators use bump pointer allocation:

This is extremely fast (O(1) allocation) but requires stack-like discipline for deallocation.

Arenas follow LIFO (last-in-first-out) cleanup:

Only the most recent allocation can be resized:

This constraint is enforced by guards in memory.c (see Development Environment chapter).

c-xrefactory uses specialized arenas for different purposes (see Components chapter for details):

When an editor buffer is modified but not yet saved:

Without preload, the server would only see the last saved version of the file. The preload mechanism ensures that:

When a file is preloaded, the server must handle reference updates carefully:

The browser stack is a linked list of OlcxReferences entries, where each entry represents a symbol lookup session:

The browser stack is populated in two stages:

This separation means that browser stack entries can become stale if files are reparsed (e.g., with preloaded content) without refreshing the stack. Users typically need to pop and re-push to get fresh reference lists after edits.

Each BrowsingMenu entry is a menu item wrapping a ReferenceableItem with UI presentation state:

Key insight: BrowsingMenu is not just a menu - it’s a menu item. A collection of BrowserMenu items forms the actual menu shown to the user.

A single browser stack entry (OlcxReferences) can contain multiple BrowsingMenu items:

This allows users to:

Browser menus are populated by scanning the referenceableItemTable:

Result: A persistent database mapping each entity (ReferenceableItem) to all its occurrences (References) across the entire codebase.

Result: Interactive navigation through the cross-reference database with selection and filtering.

This architecture separates concerns:

When currentInput runs out of lexems (read >= write), refillInputIfEmpty() uses the stream type to decide what to do:

Stream type must match buffer allocation:

Violating this invariant causes fatal errors when trying to free buffers from the wrong arena.

Pushing a NORMAL_STREAM onto the macro stack, then trying to free it as if it were MACRO_STREAM:

The Duality of Source Code

When a user edits code in Emacs, the source code exists in two forms:

For code analysis to be useful during active editing, c-xrefactory must treat editor buffers as the source of truth when they exist.

The Preloading Mechanism

When the Emacs client sends a command to the c-xref server (like PUSH or NEXT), it uses the -preload option to transmit modified buffers:

For example:

The process:

This ensures that the server analyzes what the user sees in the editor, not the potentially stale disk file.

EditorBuffer Lifecycle

To implement incremental updates, c-xrefactory tracks when each file/buffer was last parsed.

The Fields

Each FileItem in the file table has:

These are time_t values (seconds since epoch).

Dual Semantics

The lastParsedMtime field has dual semantics depending on context:

This works because editorFileModificationTime() abstracts over both:

The abstraction is seamless: code can check "has this file changed?" without caring whether it’s a disk file or editor buffer.

Change Detection

To determine if a file/buffer needs reparsing:

This pattern appears in:

C-xrefactory has two update strategies that trade off speed against completeness.

Fast Update (Default)

When used:

What it does:

Trade-off:

Example:

Full Update

When used:

What it does:

The algorithm (makeIncludeClosureOfFilesToUpdate):

Trade-off:

Example:

When Does the Difference Matter?

With modern CPUs and SSDs, the performance difference is often negligible for small to medium projects. The fast update’s header-blindness can lead to subtle bugs where changes don’t propagate. Full update is generally safer and more correct.

References flow through multiple stages in c-xrefactory, with different storage locations and ownership models at each stage.

Stage 1: Parsing

When a file is parsed:

Storage: cxMemory (a custom arena allocator)

Lifetime: Lives until the next update that reparses that file

Stage 2: The ReferenceableItemTable

The canonical storage for all references.

Key properties:

Stage 3: Session Stacks

When a user performs a PUSH operation (browsing a symbol), a session is created:

Key properties:

Why Separate Storage?

Memory ownership:

If sessions pointed directly to table references, we’d have:

Snapshots vs. live data:

The Staleness Problem

The separation causes a problem: session references can become stale.

Scenario:

Solution: processModifiedFilesForNavigation()

When NEXT/PREVIOUS operations occur, the server:

This keeps navigation working correctly with live-edited code.

Trade-offs of the Incremental Approach

Advantages:

Limitations:

For typical usage (navigating within files being actively edited), these limitations rarely matter. A fresh PUSH creates a new session with fresh references.

The key insights:

C-xref started (probably) as a cross-referencer for the languages supported (C, Java, C++), orginally had the name "xref" which became "xrefactory" when refactoring support was added. And when Mariàn released a "C only" version in 2009 most of all the "xref" references and names was changed to "c-xref". So, as most software, there is a history and a naming legacy to remember.

The source code for c-xrefactory was using a very old C style with a separate proto.h where all prototypes for all externally visible functions were placed. Definitions are all over the place and it was hard to see where data is actually declared. This must change into module-oriented include-strategy.

Of course this will have to change into the modern x.h/x.c externally visible interface model so that we get clean modules that can be unittested.

The function prototypes have been now moved out to header files for each "module". Some of the types have also done that, but this is still a work in progress.

The preprocessor macro expansion code uses arena allocators (ppmMemory, macroBodyMemory) with stack-like discipline. Arena allocators are fast (pointer bumping) but require careful lifetime management.

The Problem Pattern

Arena allocators can only resize the most recent allocation ("top-of-stack"). A common violation occurs when trying to resize a buffer after other allocations:

The correct pattern frees temporaries before growing the buffer:

Lifetime Violation Guards

The arena allocator includes guards that catch lifetime violations with detailed diagnostics:

Example: test_collation_long_expansion

This test case triggered the violation guard when macro expansion created a very large output (19 FLAG_STRING invocations). The collate() function was calling ppmFreeUntil() after copyRemainingLexems(), which needed to grow the caller’s buffer.

The fix: move ppmFreeUntil() to before copyRemainingLexems(). By this point, temporary allocations from macro expansion were already used and could be freed, allowing the buffer to become top-of-stack again.

Debugging With Guards

When a guard triggers:

The guards turn subtle crashes into clear diagnostics that point to the fix.

There are not very many unittests at this point, only covering a quarter of the code. The "units" in this project are unclear and entangled so creating unittests is hard since it was not build to be tested, test driven or even clearly modularized.

All unittests use Cgreen as the unittest framework. If you are unfamiliar with it the most important point is that it can mock functions, so you will find mock implementations of all external functions for a module in a corresponding <module>.mock file.

Many modules are at least under test, meaning there is a <module>_tests.c in the unittest directory. Often only containing an empty test.

In the tests directory you will find tests that exercise the external behaviour of c-xref, "acceptance tests" or "system tests". Some tests actually do only that, they wouldn’t really count as tests as there are no verification except for the code being executed.

There are two basic strategies for the tests:

Some tests do not even test its output in any meaningful way and only provide coverage.

Some tests do a very bad job at verifying, either because my understanding at that time was very low, or because it is hard to verify the output. E.g. a "test" for generating references might only grepping the CXrefs files for some strings, not verifying that they actually point to the correct place.

Hopefully this will change as the code gets into a better state and the understanding grows.

Tests live in the tests directory and are auto-discovered by name: any directory starting with test_ will be recognized as a test case.

Each test typically includes:

Most tests use tests/Makefile.boilerplate which provides common macros:

The key macros are:

When $(VERIFY) passes, the output file is removed. This means you can easily identify failing tests by looking for test directories that still contain an output file. The utils/failing script lists these.

To suspend a test (skip it during test runs), create a .suspended file in the test directory.

Since all(?) c-xref operation rely on an options file which must contain absolute file paths (because the server runs as a separate process) it must be generated whenever the tests are to be run in a different location (new clone, test was renamed, …​).

This is performed by using a common template in tests and a target in tests/Maefile.boilerplate.

Each test should have a clean target that removes any temporary and generated files, including the .c-xrefrc file and generated references. This way it is easy to ensure that all tests have updated .c-xrefrc files.

Since many operations are performed from the editor, and the editor starts an "edit server" process, many tests need to emulate this behaviour.

The edit server session is mostly used for navigation. Refactorings are actually performed as separate invocations of c-xref.

In utils there is a server_driver.py script, which will take as input a file containing a sequence of commands. You can use this to start an edit, refactory or reference server session and then feed it with commands in the same fashion as an editor would do. The script also handles the communication through the buffer file (see [Editor Interface](./Design:-Editor-Interface)).

You can relatively easy re-create a sequence of interactions by using the sandboxed Emacs in tests/sandboxed_emacs.

There are two ways to use it, "make spy" or "make pure". With the "spy" an intermediate spy is injected between the editor and the edit server, capturing the interaction to a file.

With "pure" you just get the editor setup with c-xref-debug-mode and c-xref-debug-preserve-tmp-files on. This means that you can do what ever editor interactions you want and see the communication in the *Messages* buffer. See [Editor Interface](./Design:-Editor-Interface) for details.

Once you have figure out which part of the *Messages* buffer are interesting you can copy that out to a file and run utils/messages2commands.py on it to get a file formatted for input to server_driver.py.

utils/covers.py is a Python script that, in some enviroments, can list which test cases execute a particular line.

This is handy when you want to debug or step through a particular part of the code. Find a test that covers that particular line and run it using the debugger (usually make debug in the test directory).

Synopsis:

utils/sandboxed starts a sandboxed Emacs that uses the current elisp code and the c-xref from src. This allows you to run a test changes without having to polute your own setup.

This actually runs the tests/sandboxed_emacs pure version, which also sets up a completely isolated Emacs environment with its own packages loaded, configuration etc. See below.

Synopsis:

In-memory references become the single source of truth. Currently, the system has two sources (in-memory referenceableItemTable and .cx files on disk) with different code paths for navigation vs. refactoring. This creates complexity, bugs, and makes preloaded editor buffers work inconsistently.

The following diagram shows how the remaining architectural pieces depend on each other. Work proceeds from bottom to top.

Both navigation and refactoring already use the same in-memory ReferenceableItemTable (see Chapter 18: Unified In-Memory Table Discovery for details). The infrastructure for memory-as-truth is largely already in place:

The path forward is:

Once memory is truth, the .cx file is a startup snapshot — a serialized copy of previously-computed references that avoids a full project parse on server start.

Startup sequence:

After startup, the .cx file is not consulted. All operations read from the in-memory table.

Writing the snapshot:

The snapshot is written from disk-file-derived references only. Modified-buffer data must never be persisted — the snapshot reflects disk state, so that startup validation (step 2) remains correct.

Cold start (no .cx file): All project source files are parsed. This is slower but self-healing — no manual action needed to recover from a missing or corrupt snapshot.

Enable VS Code, Neovim, and other LSP-capable editors to use c-xrefactory’s refactoring and navigation features with full feature parity to the Emacs integration.

The native Emacs path has a mature sync/execute separation (ADR 20): before each operation, reparse stale files in three passes, then execute with a current in-memory table. The LSP protocol provides the same separation naturally — document lifecycle events (didOpen, didChange, didSave) are sync points, and requests (definition, references) are operations that assume the table is current.

This means the LSP path does not need its own architecture. It can reuse the same initialization and reparsing infrastructure as the native path, triggered by different events.

Once the foundation above is in place, the following LSP features become possible:

Steps 1-3 above are independent of the "Memory as Truth" convergence — they reuse existing infrastructure as-is. Full feature parity (rename, extract) additionally depends on:

Provide refactoring capabilities that understand C semantics, going beyond what generic text-based tools can offer.

See Chapter 16: Move Function Between Files for implementation details and phase breakdown.

Reorganize code by moving function definitions between source files while maintaining correctness. Essential for refactoring large codebases into better module structure.

Example: Extract a utility function from main.c to utils.c, automatically handling visibility changes and header declarations.

Remove extern declaration: Remove the moved functions extern declaration from the header file for the source, if the function was not static.

Include management: Automatically add necessary #include directives based on dependencies.

Helper function detection: Identify and optionally move static helper functions that the moved function depends on. Prevents broken builds.

Smarter header placement: Automatically find the right location in header files based on existing declarations and dependencies.

Preview: Show what will be changed before applying the refactoring.

Language Server Protocol (LSP) integration enables c-xrefactory to work with modern editors and IDEs (VS Code, vim/neovim with LSP plugins, etc.) beyond Emacs. This opens c-xrefactory’s refactoring and navigation capabilities to a much wider audience.

Go to Definition (textDocument/definition):

File Parsing:

Single-file scope: Only symbols in the currently opened file are accessible. To navigate to a function in another file, you must first open that file.

No local variables: Go-to-definition doesn’t work for local variables or function parameters. This works in Emacs mode but requires architectural changes for LSP.

Missing LSP features: Only textDocument/definition is implemented. Coming features:

See the README for LSP client configuration examples. Basic setup:

When the server does a cold start (no disk database), it parses all discovered compilation units to populate in-memory references. During this parsing, warnings and errors may occur (e.g., missing include files, syntax errors from missing -D defines). These are expected and harmless — c-xrefactory continues best-effort — but users benefit from seeing them to tune their .c-xrefrc (adding -I paths, -D defines, etc.).

The legacy -create command in Emacs offered "View log file?" after completion, showing a file with all parsing messages. The cold start server path needs an equivalent.

There are also other situations when a re-parse or re-discovery might throw the same kind of errors.

The current protocol has no non-modal message channel. Every message type either shows a modal dialog (PPC_WARNING, PPC_ERROR, PPC_INFORMATION) requiring "Press a key to continue", or writes a transient line to the minibuffer (PPC_BOTTOM_INFORMATION). Sending per-file warnings during cold start floods the user with one modal per warning, effectively locking up the editor.

Option A: New protocol record type (recommended) — Add a PPC_LOG record that the Emacs client silently appends to a c-xref-log buffer. After cold start completes, send a PPC_BOTTOM_INFORMATION saying "Indexing done: N warnings (see c-xref-log)". Clean separation, no modals, inspectable at leisure. Touches protocol definition, C server code, and Emacs client.

Option B: Server-side batching — Collect all warnings during cold start into a single string, send as one PPC_INFORMATION after parsing completes. One modal instead of many. Simpler but still modal.

Option C: Summary only — Send a PPC_BOTTOM_WARNING summary to the minibuffer (e.g., "3 files could not be opened during indexing"). Details go only to the log file. Minimal change but no inspectable buffer in Emacs.

The #include preprocessor statments are referenceable items just like typenames and variables. That means it is logical to expect that placing the cursor on an #include line and navigation would follow the same UX as for a variable: PUSH moves cursor to the definition (in this case the file itself) and NEXT to the next "reference", which in this case would be the next #include statement for that file.

This does not happen.

This needs to be investigated and understood before an attempt to design a solution can be made.

The cxfile module was designed when the disk database was the source of truth. The .cx files were not just a cache — they were the reference database. Operations like "show all references to symbol X" or "find unused symbols" were implemented as table-driven scans over the .cx file format: hash the symbol name to find the right partition, stream through it, apply a per-operation filter callback. This was efficient for the batch cross-referencer that c-xrefactory evolved from, where the workflow was c-xref -create → c-xref -update → query the on-disk database.

With the shift toward memory-as-truth (Chapter 15: Roadmap), the in-memory referenceableItemTable is becoming the authoritative source, and .cx files are becoming a startup snapshot. But cxfile.c still carries the old design: it combines disk I/O with operation-specific filtering logic, and several operations still bypass the in-memory table to read from disk directly.

The public interface (cxfile.h) reflects this mixed heritage:

The four scan* functions read directly from disk .cx files and merge results into the in-memory referenceableItemTable. Each encodes a specific use case (menu creation, macro completion, unused detection, symbol search) that a storage module shouldn’t know about.

The ensureReferencesAreLoadedFor function is the right pattern: load from disk into memory, then let the caller query memory. The scan* functions bypass this by combining load + filter in one step.

With the memory-as-truth direction (Chapter 15), cxfile.c is already shrinking in importance:

These are independent improvements, not a phased plan:

Make cxFileHashNumberForSymbol static. It’s a partitioning implementation detail. Only called internally and from one external site that can be rerouted.

Make searchSymbolCheckReference static. Move the search matching logic to the caller (search.c already exists). The storage layer shouldn’t know about search string matching.

Replace scanReferencesToCreateMenu and scanForMacroUsage with in-memory queries. Both load a symbol’s references from disk and then filter them. With entry refresh populating the in-memory table, these should query referenceableItemTable directly — the data is already there. ensureReferencesAreLoadedFor can serve as the fallback if the symbol hasn’t been loaded yet.

Convert scanForGlobalUnused and scanForSearch to visitor pattern. These need to scan all symbols, not just one — a bulk operation that makes sense as a generic scanAllReferences(visitor, context) function. The visitor decides what to do with each item (check for unused, match search pattern). The operation-specific logic moves to the callers.

The yylex.c file is 2353 lines and combines multiple responsibilities:

The macro expansion code is a substantial, cohesive subsystem that would benefit from extraction into its own module. Currently, it’s deeply embedded in yylex.c, making both lexing and macro expansion harder to understand and test in isolation.

The macro expansion system in yylex.c comprises:

Core Responsibilities (~800 lines)

Key State

Memory Lifetime Separation

The system uses two distinct memory arenas with different lifetimes:

This separation is fundamental and should be preserved in any refactoring.

Extract macro expansion into a new module: macroexpansion.c/h

Public Interface

The new module would expose a minimal, focused API:

Module Boundaries

What moves to macroexpansion.c:

What remains in yylex.c:

Dependencies:

The macro module would depend on:

Architectural

Maintainability

Future flexibility

Phase 1: Preparation (Already Complete)

✓ Create LexemBufferDescriptor type for buffer management
✓ Refactor buffer expansion functions to use descriptor
✓ Eliminate return values for size updates

Phase 2: Create Module Structure

Phase 3: Incremental Function Migration

Move functions in this order (lowest risk first):

Phase 4: Integration and Cleanup

Risk: Complex dependencies

Mitigation:

Risk: Performance overhead

Mitigation:

Assessment: Low risk - macro operations are complex enough that function call overhead is negligible

Risk: Breaking existing tests

Mitigation:

Refactoring operations that need structural information from parsing (function boundaries, move target validation, parameter positions, etc.) do not call the parser directly. Instead, they go through parseBufferUsingServer() — a function that constructs magic string arguments (like "-olcxmovetarget", "-olcxgetfunctionbounds") and runs a full initServer() + callServer() cycle. This is the internal equivalent of the editor sending a command over the protocol.

This makes the control flow circular and hard to follow: a refactoring operation (already running inside the server) constructs fake arguments, re-enters the server, which dispatches the operation, which calls the parser, which produces a side-effect in global state, which the original caller then reads.

A clean parser API in parsing.h/c already exists and handles all call sites in server.c:

Two convenience functions have clean APIs but still bridge internally:

These demonstrate the migration pattern: the public API is already clean, only the implementation still bridges.

9 bridge calls to eliminate (2 in parsing.c, 7 in refactory.c):

Each call follows the same pattern: set global flags → call parseBufferUsingServer → read results from global state. The migration for each is: set ParsingConfig fields → call parser directly → read results from the same global state.

The two parsing.c bridges show the pattern clearly. Today:

The ParsingConfig is already set up — the bridge call is redundant, it just re-enters the server which re-reads the same config. Replacing it with a direct callParser() (similar to parseToCreateReferences) is the straightforward next step.

The refactory.c calls are slightly more involved because they also need buffer preloading (the editor marker’s buffer must be set as input for parsing) and some set additional global flags. But the mechanism is the same.

This refactoring is independent of the memory-as-truth architecture. The bridge removal is about how refactoring operations invoke parsing internally — not about where references come from or how they’re stored. It can proceed at any time, one call site at a time.

The codebase has two hash table implementations generated by C macro templates:

Only two tables still use hashtab:

All other tables (editorBufferTable, symbolTable, referenceableItemTable) already use hashlist.

The lightweight file structure scanning work (ADR 22) exposed hashtab’s limitations:

Migrate fileTable and macroArgumentTable from hashtab to hashlist, then remove hashtab entirely.

fileTable migration:

macroArgumentTable migration:

Remove hashtab:

The file table serves a dual purpose: hash lookup by name AND array indexing by file number. File numbers are used throughout the codebase as compact identifiers (stored in references, positions, sessions, etc.).

Hashtab provides both naturally — the hash slot index is the file number. Hashlist doesn’t — elements live in chains, not at stable array positions.

Possible approaches:

The parallel array approach is likely cleanest — it separates the two concerns (name lookup vs number lookup) explicitly.

The editor.c module (630+ lines) bundles at least four unrelated responsibilities under the misleading name "editor". The name suggests it’s about the external editor (Emacs), but most of the code has nothing to do with editor integration — it’s internal text manipulation infrastructure used by refactoring.

1. Text memory allocator (editorMemory[], allocateNewEditorBufferTextSpace, freeTextSpace)

A power-of-2 block allocator for buffer text content. Custom malloc pool with free lists indexed by size class. No relationship to "editing" — this is a memory management concern.

2. In-buffer text editing (replaceStringInEditorBuffer, moveBlockInEditorBuffer, removeBlanksAtEditorMarker)

The refactoring engine’s hands — insert, delete, replace, and move text within buffers, with undo tracking and marker adjustment. This is the actual "editing" code, but it operates on the server’s internal buffers, not the external editor.

3. Reference/marker coordinate conversion (convertReferencesToEditorMarkers, convertEditorMarkersToReferences)

Bridges two coordinate systems: the reference world (file number, line, column) and the buffer world (buffer pointer, byte offset). Walks buffer text to convert line/col to offset and vice versa. Used by refactoring to go from "where are the references" to "where do I edit".

4. Content buffer lifecycle (loadFileIntoEditorBuffer, loadAllOpenedEditorBuffers, closeAllEditorBuffers, quasiSaveModifiedEditorBuffers, editorFileModificationTime)

Buffer loading, closing, quasi-save (mtime bookkeeping for refactoring), and file-stat wrappers that check buffers before falling back to filesystem. This is the content buffer management that belongs with the ContentBuffer type.

The "Editor" prefix pervades the codebase beyond just this module:

None of these are about the external editor. They’re the server’s internal text manipulation primitives. TextMarker, TextRegion, TextUndo would be more accurate names, but renaming has cascading effects throughout the codebase.

editorInit() would move to wherever initialization belongs (it currently just calls initEditorBufferTable()).

editorMapOnNonExistantFiles is a completion helper that iterates all buffers to find files existing in buffers but not on disk. It belongs with completion or content buffer management.

Server operations use the prefix olcx (likely "on-line cross-references"), a legacy abbreviation that conveys no meaning to anyone reading the code or protocol today. Option names like -olcxpush, -olcxtagsearch, -olcxgetprojectname are opaque compared to what they could be: -push, -search, -get-project-name.

The word "tag" in search-related names (-olcxtagsearch, c-xref-search-in-tag-file, c-xref-tag-results-buffer) is a leftover from ctags and friends. There is no ctags compatibility — the term is purely misleading. The two menu entries "Search Definition in Tags" and "Search Symbol" both trigger the same OP_SEARCH operation, differing only in a filter flag (-searchdef). The word "tag" suggests they are fundamentally different, when they are not.

The rename touches three layers:

Protocol options (server C code + Emacs client): ~60 operations prefixed with -olcx* in options.c. These become plain descriptive names: -push, -pop, -goto, -complete, -search, -rename, -extract, -filter, -get-project-name, etc.

Emacs function and variable names: Functions like c-xref-search-in-tag-file, c-xref-get-tags, c-xref-interactive-tag-search-* and variables like c-xref-tag-results-buffer, c-xref-tag-search-mode-map.

System tests: Every commands.input file uses these option names. Mechanical update.

Shortest clear name: Prefer -push over -browse-push. Stack operations (push, pop) are domain concepts — without understanding them you can’t understand browsing anyway. No grouping prefix needed.

Two kinds of filter: The browsing UI has two filter concepts that should remain distinct:

These could become -reference-filter and -symbol-filter for extra clarity, but -filter and -menu-filter are adequate.

No compatibility shim needed: Distribution is via git repo updates, and the Emacs upgrade menu entry kills the server, reloads elisp, and starts a new server on the next operation. Old and new never need to coexist.

Support both old and new names in the option parser temporarily (two strcmp entries per option). Rename everywhere else (client, tests, docs). Remove the old names in a later commit. This eliminates any breakage window.

Low urgency, high value for readability. Can be done incrementally — start with the "tag" removal (search-related names only), then expand to the full olcx rename later.

Symbol navigation (PUSH/NEXT/PREVIOUS/POP) merges references from two sources:

When you PUSH on a symbol, the navigation menu creation (createSelectionMenuForOperation) does:

This dual-source approach allows navigation without full project parse while providing updated positions for modified files.

MUST be maintained:

RefactoryMode runs XrefMode as an internal sub-task to update the reference database before performing refactorings.

This is critical to understand because:

When a refactoring is invoked:

The Emacs client spawns a separate RefactoryMode process rather than using the interactive server because:

When implementing staleness detection for navigation (refreshing references when files are modified), we added an optimization to server.c:

This optimization prevents navigation from thinking a file is "stale" immediately after parsing it with preloaded content.

This broke refactoring operations (test_delete_parameter_with_preload). The problem:

The architecture has two sources of references:

The lastParsedMtime optimization conflated these: * It correctly told navigation "in-memory is current" * But incorrectly told xref "persistent database is current" (when it wasn’t)

Exclude refactoring operations from the optimization:

This allows: * Navigation to use the optimization (in-memory is sufficient) * Refactoring to trigger xref re-indexing (reads preload, updates .cx correctly)

This workaround highlights the fundamental tension: navigation and refactoring use different sources of truth. The proper fix is ADR-0014's unified on-demand architecture where both use the same in-memory reference database, eliminating the need for callXref() entirely.

When navigating (PUSH/NEXT), the server detects if the current file is "stale" by comparing:

If buffer→modificationTime > lastParsedMtime, the file is considered stale and refreshStaleReferencesInSession() is called.

refreshStaleReferencesInSession() in cxref.c must preserve cross-file references while updating references from the modified file:

Key insights:

During PUSH, createSelectionMenu creates menu items and adds them to sessionEntry→menu. These items are marked as selected. When you NEXT, the menu’s reference list is what gets used for navigation (via recomputeSelectedReferenceable).

If during refresh we: 1. Clear menu refs 2. Scan from disk

The scan calls createSelectionMenu which in turn calls addReferenceableToBrowsingMenu. This function compares incoming items by (linkName, includeFileNumber, type). If the includeFileNumber from the disk scan differs from the existing menu item’s value, a new menu item is created with selected=false.

The refs are then added to this new unselected item, while the original selected item has empty refs. recomputeSelectedReferenceable only processes selected items, so navigation loses most of its references.

This is a manifestation of the "two sources of truth" architecture. The long-term fix (per roadmap) is to parse all project files at startup, keeping complete references in memory. Then stale refresh would just re-parse the single file and the in-memory table would already have all cross-file references.

Implemented. TBD.

Implemented. TBD.

Implemented. TBD.

Purpose: Convert functions that are only used within their compilation unit to static storage class for better encapsulation and compiler optimization.

When to use:

Input:

Availability:

When cursor is on a non-static function definition where all callers are in the same file.

Algorithm:

Output:

Example:

Benefits:

Notes:

Input:

Algorithm:

Output:

Notes:

Purpose: Rename a file appearing in an include and update all the include directives

When to use:

Input:

Availability

When the cursor is on an #include directive. The file it references will be the "source".

Algorithm:

Output:

See Chapter 16a: Planned Refactoring Features for detailed design and implementation status.

Proposed refactoring to move a function definition from one C source file to another while automatically managing visibility (static vs extern) and potentially adding necessary declarations and includes.

Status: Phase 1 MVP complete.

Tentative.

Tentative.

Purpose: Convert functions that are only used within their compilation unit to static storage class for better encapsulation and compiler optimization.

When to use:

Input:

Availability:

When cursor is on a non-static function definition where all callers are in the same file.

Algorithm:

Output:

Example:

Benefits:

Notes:

Static memory (SM_ prefix) are static areas allocated by the compiler which is then indexed using a similarly named index variable (e.g. ftMemory and ftMemoryIndex), something the macros took advantage of. These are

One special case of static memory also exist:

These areas cannot be extended, when it overruns the program stops.

Cache Point Model: The system placed strategic snapshots of parser state at external definition boundaries (functions, global variables, etc.). When files were re-processed, the system could validate cache integrity, recover from cache points, and resume parsing only from the first changed definition onward.

Separation of Concerns: Recent refactoring had separated file tracking from cache validation:

Cache Point Management (caching.c):

File Modification Tracking:

The FileItem structure maintained multiple timestamp fields for tracking file modification:

Input Stream Caching:

C Parser Integration: Both C and Yacc parsers placed cache points after each external_definition, but only when not processing include files (includeStack.pointer == 0).

Parser-Specific Behavior:

The caching system was deeply integrated throughout the parsing pipeline:

Cache Hit Scenarios:

Optimization Benefits:

Once the FILL-macros was removed, we could move the enum-generation to use the actual c-xref. So from now on we build c-xref directly from the sources in the repo. Changes to any enums will trigger a re-generation of the enumTxt-files but since the enumTxt-files are only conversion of enum values to strings any mismatch will not prevent compilation, and it would even be possible to a manual update. This is a big improvement over the previous situation!

As indicated in FILL macros the bootstrapping of FILL-macros has finally and fully been removed.

Gone is also the compiler_defines.h, which was just removed without any obvious adverse effects. Maybe that will come back and bite me when we move to more platforms other than linux and MacOS…​

Left is, at this point, only the enumTxt generation, so most of the text below is kept for historical reasons.

c-xref uses a load of structures, and lists of them, that need to be created and initialized in a lot of places (such as the parsers). To make this somewhat manageable, c-xref itself parses the strucures and generates macros that can be used to fill them with one call.

c-xref is also bootstrapped into reading in a lot of predefined header files to get system definitions as "preloaded definitions".

Why this pre-loading was necessary, I don’t exactly know. It might be an optimization, or an idea that was born early and then just kept on and on. In any case it creates an extra complexity building and maintaining and to the structure of c-xref.

So this must be removed, see below.

The bootstrapping uses c-xref's own capability to parse C-code and parse those structures and spit out filling macros, and some other stuff.

This is done using options like `-task_regime_generate' which prints a lot of data structures on the standard output which is then fed into generated versions of strFill, strTdef(no longer exists) and enumTxt by the Makefile.

The process starts with building a c-xref.bs executable from checked in sources. This compile uses a BOOTSTRAP define that causes some header files to include pre-generated versions of the generated files (currently strFill.bs.h and enumTxt.bs.h) which should work in all environments.

After the c-xref.bs has been built, it is used to generate strFill and enumTxt which might include specific structures for the current environment.

HOWEVER: if FILL macros are used for structures which are different on some platforms, say a FILE structure, that FILL macro will have difference number of arguments, so I’m not sure how smart this "smart" generation technique actually is.

TODO: Investigate alternative approaches to this generate "regime", perhaps move to a "class"-oriented structure with initialization functions for each "class" instead of macros.

In options.h there are a number of definitions which somehow are sent to the compiler/preprocessor or used so that standard settings are the same as if a program will be compiled using the standard compiler on the platform. At this point I don’t know exactly how this conversion from C declarations to compile time definitions is done, maybe just entered as symbols in one of the many symboltables?

Typical examples include "__linux" but also on some platforms things like "fpos_t=long".

I’ve implemented a mechanism that uses "gcc -E -mD" to print out and catch all compiler defines in compiler_defines.h. This was necessary because of such definitions on Darwin which where not in the "pre-programmed" ones.

TODO?: As this is a more general approach it should possibly completely replace the "programmed" ones in options.c?

To be able to print the string values of enums the module generate.c (called when regime was RegimeGenerate) could also generate string arrays for all enums. By replacing that with some pre-processor magic for the few that was actually needed (mostly in log_trace() calls) we could do away with that whole "generate" functionality too.

(Last commit with enum generation intact is https://github.com/thoni56/c-xrefactory/commit/aafd7b1f813f2c17c684ea87ac87a0be31cdd4c4.)

For some cases the string representing the value of an Enum is needed. c-xref handles this using the "usual" 'parse code and generate' method. The module generate.c does this generation too.

Also in options.h some standard-like include paths are added, but there is a better attempt in getAndProcessGccOptions() which uses the compiler/preprocessor itself to figure out those paths.

TODO?: This is much better and should really be the only way, I think.

Since at bootstrap there must exist FILL-macros with the correct field names this strategy is an obstacle to cleaning up the code since every field is referenced in the FILL macros. When a field (in a structure which are filled using the FILL macro) changes name, this will make initial compilation impossible until the names of that field is also changed in the strFill.bs.h file.

One way to handle this is of course to use c-xrefactory itself and rename fields. This requires that the project settings also include a pass with BOOTSTRAP set, which it does.

I’ve started removing this step. In TODO.org I keep a hierarchical list of the actions to take (in a Mikado kind of style).

The basic strategy is to start with structures that no other structure depends on. Using the script utils/struct2dot.py you can generate a DOT graph that shows those dependencies.

Removal can be done in a couple of ways

The strTdef.h was generated using the option -typedefs as a part of the old -task_regime_generate strategy and generated typedef declarations for all types found in the parsed files.

I also think that you could actually merge the struct definition with the typedef so that strTdef.h would not be needed. But it seems that this design is because the structures in proto.h are not a directed graph, so loops makes that impossible. Instead the typedefs are included before the structs:

This is now ideomatically solved using the structs themselves: