C-xrefactory - a C/Yacc refactoring and code browsing tool - Design Documentation

The LSP protocol was designed for single-shot, non-interactive operations. This creates constraints for c-xrefactory’s advanced refactorings:

Interactive Refactorings: C-xrefactory’s extract/parameter operations require multi-step user input (names, positions, declarations). LSP’s textDocument/codeAction doesn’t support interactive dialogs.

Symbol Browsing: C-xrefactory provides interactive symbol browsers with filtering and keyboard navigation. LSP returns flat reference lists with no standard for interactive UI.

Strategy: The LSP implementation aims to:

LSP serves to make c-xrefactory more accessible while the Emacs client probably will remain the gateway to its complete refactoring power.

cxrefCore is the core container. It does all the work when it comes to finding and reporting references to symbols, communicating refactoring requests as well as storing reference information for longer term storage and caching.

Although c-xref can be used as a command line tool, which can be handy when debugging or exploring, it is normally used in "server" mode. In server mode the communication between the editor extension and the cxrefCore container is a back-and-forth communication using a non-standard protocol over standard pipes.

The responsibilities of cxrefCore can largely be divided into

At this point it seems like refactorings are performed as separate invocations of c-xref rather than through the server interface.

The EditorExtension container is responsible for plugging into an editor of choice and handle the user interface, buffer management and executing the refactoring edit operations.

Currently there is only one such extension supported, for Emacs, although there existed code, still available in the repo history, for an extension for jEdit which hasn’t been updated, modified or checked for a long time and no longer is a part of this project.

The References database stores crossreferencing information for symbols visible outside the module it is defined in. Information about local/static symbols are not stored but gathered by parsing that particular source file on demand.

Currently this information is stored in a somewhat cryptic, optimized text format.

This storage can be divided into multiple files, probably for faster access. Symbols are then hashed to know which of the "database" files it is stored in. As all crossreferencing information for a symbol is stored in the same "record", this allows reading only a single file when a symbol is looked up.

Using an external preprocessor would mean that all macros would be expanded before c-xrefactory sees the code. This would make it impossible to:

By parsing the source at the macro level, c-xrefactory operates as a source-level tool rather than a post-preprocessed tool. This allows it to work with the code as developers see and write it, preserving all macro information for navigation and refactoring.

The integrated preprocessor is implemented in yylex.c and includes:

The integrated preprocessor does not support all modern preprocessor features:

These limitations mean that c-xrefactory may report syntax errors when encountering modern platform headers that use these features, even though the code compiles correctly with standard compilers.

This design choice represents a fundamental trade-off:

The benefit of macro-level refactoring is considered more valuable than perfect preprocessor compatibility, as it is a key differentiator for c-xrefactory.

Lexing/scanning is performed in two layers, one in lexer.c which seems to be doing the actual lexing into lexems which are put in a lexembuffer. This contains a sequence of encoded and compressed symbols which first has a LexemCode which is followed by extra data, like Position. These seems to always be added but not always necessary.

The higher level "scanning" is performed, as per ususal, by yylex.c. lexembuffer defines some functions to put and get lexems, chars (identifiers and file names?) as well as integers and positions.

At this point the put/get lexem functions take a pointer to a pointer to chars (which presumably is the lexem stream in the lexembuffer) which it also advances. This requires the caller to manage the LexemBuffer’s internal pointers outside and finally set them right when done.

It would be much better to call the "putLexem()"-functions with a lexemBuffer but there seems to be a few cases where the destination (often dd) is not a lexem stream inside a lexemBuffer. These might be related to macro handling.

As the refactoring functions need some amount of semantic information, in the sense of information gathered during parsing, this information is collected in various ways when c-xref calls the "sub-task" to do the parsing required.

Two structures hold information about various things, among which are the memory index at certain points of the parsing. Thus it is possible to verify e.g. that a editor region does not cover a break in block or function structure. This structure is, at the point of writing, called parsedInfo and definitely need to be tidied up.

c-xrefactory’s core functionality relies on a sophisticated symbol database that stores cross-references, definitions, and usage information for all symbols in a project. The database is implemented as a collection of binary .cx files and supports:

The symbol database uses a hash-partitioned file structure:

Each .cx file contains structured records with specific keys:

The symbol lookup process follows this pipeline:

The current implementation of the reference database is file based, with an optimized storage format.

There is limited support to automatically keep these updated during an edit-compile cycle, you might have to update manually now and then.

The project settings (or command line options) indicate where the file(s) are created and one option controls the number of files to be used, -refnum.

This file (or files) contains compact, but textual representations of the cross-reference information. Format is somewhat complex, but here are somethings that I think I have found out:

So a line might say

The line identifies the file with id 12205. The file was last included in an update of refs at sometime which is identified by 1522108169 (mtime), has not been part of a full update of xrefs, was mentioned on the command line. (I don’t know what the 'a' means…​) Finally, the file name itself is 84 characters long.

All information about an externally visible symbol is stored in one, and only one reference file, determined by hashing the linkname of the symbol. So it will always suffice to read one reference file when consulting the reference database (in the form of CXFILE) for a symbol.

The reading of the CXFILE format is controlled by `scanFunctionTable`s. These consists of a list of entries, one for each key/tag/recordCode (see format description above) that the scan will process.

As the reference file reading encounters a key/tag/recordCode it will consult the table and see if there is an entry pointing to a handler function for that key/tag/recordCode. If so, it will be called.

The editor invokes a new c-xref process for the following cases:

There is some magical editor buffer management happening inside of c-xref which is not clear to me at this point. Basically it looks like the editor-side tries to keep the server in sync with which buffers are opened with what file…​

At this point I suspect that -preload <file1> <file2> means that the editor has saved a copy of <file1> in <file2> and requests the server to set up a "buffer" describing that file and use it instead of the <file1> that recides on disk.

This is essential when doing refactoring since the version of the file most likely only exists in the editor, so the editor has to tell the server the current content somehow, this is the -preload option.

The initial invocation of the edit server creates a process with which communication is over stdin/stdout using a protocol which from the editor is basically a version of the command line options.

When the editor has delivered all information to the server it sends 'end-of-option' as a command and the edit server processes whatever it has and responds with <sync> which means that the editor can fetch the result in the file it named as the output file using the '-o' option.

Setting the emacs variable c-xref-debug-mode forces the editor to copy the content of such an output file to a separate temporary file before re-using it.

For some interactions the editor starts a completely new and fresh c-xref process, see below. And actually you can’t do refactorings using the server, they have to be separate calls. (Yes?) I have yet to discover why this design choice was made.

Communication between the editor and the server is performed using text through standard input/output to/from c-xref. The protocol is defined in src/protocol.tc and must match editor/emacs/c-xrefprotocol.el.

The definition of the protocol only caters for the server→editor part, the editor→server part consists of command lines resembling the command line options and arguments, and actually is handled by the same code.

The file protocol.tc is included in protocol.h and protocol.c which generates definitions and declarations for the elements through using some macros.

There is a similar structure with c-xrefprotocol.elt which includes protocol.tc to wrap the PROTOCOL_ITEMs into defvars.

There is also some Makefile trickery that ensures that the C and elisp impementations are in sync.

One noteable detail of the protocol is that it carries strings in their native format, utf-8. This means that lengths need to indicate characters not bytes.

The editor fires up a server and keeps talking over the established channel (elisp function 'c-xref-start-server-process'). This probably puts extra demands on the memory management in the server, since it might need to handle multiple information sets and options (as read from a .cxrefrc-file) for multiple projects simultaneously over a longer period of time. (E.g. if the user enters the editor starting with one project and then continues to work on another then new project options need to be read, and new reference information be generated, read and cached.)

FINDINGS:

The editor server is started using the appropriate command line option and then it keeps the communication over stdin/stdout open.

The editor part sends command line options to the server, which looks something like (from the read_xrefs test case):

In this case the "-olcxpush" is the operative command which results in the following output

As we can see from this interaction, the server will handle (all?) input as a command line and manage the options as if it was a command line invocation.

This explains the intricate interactions between the main program and the option handling.

The reason behind this might be that a user of the editor might be editing files on multiple projects at once, so every interrogation/operation needs to clearly set the context of that operation, which is what a user would do with the command line options.

It seems that all on-line editing server functions have an olcx prefix, "On-Line C-Xrefactory", maybe…​

One thing that really confused me in the beginning was that the editor, primarily Emacs, don’t use the actual server that it has started for refactoring operations (and perhaps for other things also?). Instead it creates a separate instance with which it talks to about one refactoring.

I’ve just managed to create the first automatic test for refactorings, olcx_refactory_rename. It was created by running the sandboxed emacs to record the communication and thus finding the commands to use.

Based on this learning it seems that a refactoring typically is a single invocation of c-xref with appropriate arguments (start & stop markers, the operation, and so on) and the server then answers with a sequence of operations, like

I haven’t investigated the internal flow of such a sequence, but it is starting to look like c-xref is internally re-reading the initialization, I’m not at this point sure what this means, I hope it’s not internal recursion…​

Each type of refactoring has it’s own little "language". E.g. extracting a method/function using -refactory -rfct-extract-function will return something like

So there is much logic in the editor for this. I suspect that the three <str> parts are

If this is correct then all extractions copy the region verbatim and then the server only have to figure out how to "glue" that to a semantically correct call/argument list.

As a side note the editor asks for a new name for the function and then calls the edit server with a rename request (having preloaded the new source file(s) of course).

Dechiffrering the interaction between an editor and the edit server in c-xrefactory isn’t easy. The protocol isn’t very clear or concise. Here I’m starting to collect the important bits of the invocation, the required and relevant options and the returned information.

The test cases for various refactoring operations should give you some more details.

All of these require a -p (project) option to know which c-xref project options to read.

Memory allocation is managed through the Memory structure, which implements an arena/bump allocator. Different memory arenas serve different purposes:

See Modules chapter for detailed description of each arena’s purpose and lifetime.

The optMemory arena requires special handling because Options structures are copied during operation. When copying, all pointers into option memory must be adjusted to point into the target structure’s memory area, not the source’s.

Functions like copyOptions() perform this pointer adjustment through careful memory arithmetic, traversing a linked list of all memory locations that need updating.

There are three possible sources for options.

Not all options are relevant in all cases.

All options sources uses exactly the same format so that the same code for decoding them can be used.

When the editor has a file open it needs to "belong" to a project. The logic for finding which is very intricate and complicated.

In this code there is also checks for things like if the file is already in the index, if the configuration file has changed since last time, indicating there are scenarios that are more complicated (the server, obviously).

But I also think this code should be simplified a lot.

The yylex module has the standard interface required by any yacc-based parser, which is a simple yylex(void) function.

The Memory module provides arena-based allocation for performance-critical and request-scoped operations:

Key functions (see memory.h):

See the "Arena Allocator Lifetime Violations" section in the Development Environment chapter for:

Modern systems have abundant virtual memory. Possible improvements:

The experimental FlushableMemory type explores some of these ideas but hasn’t replaced current implementation.

Read and write the CXref database in "plain" text format.

The current file format for the cross-reference data consists of records with the general format

There are two important types of lines, a file information line and a symbol information line.

The actual keys are documented in cxfile.c, but here is an example file information line:

First we have two simple value/key pairs. We see "32571f" indicating that this is file information for file with file number 32571.

Secondly we have "1715027668m". This is the modification time of the file which is stored to be able to see if that file has been updated since the reference database was last written.

And the third part is "21:/usr/include/ctype.h", which is of a record type that is a bit more complex. The number is the length of the value. The ':' indicates that the record is a filename.

A ReferenceableItem represents a referenceable entity in the codebase - something that can be referenced from multiple locations:

Each ReferenceableItem contains:

ReferenceableItems are stored in the referenceableItemTable (hash table) and persisted to .cx files.

A Reference represents a single occurrence of a ReferenceableItem at a specific location:

Each ReferenceableItem maintains a linked list of all its References, allowing you to find every place that entity appears in the codebase.

Note: The term "Reference" in this context means "occurrence" - one specific use of an entity at one location. This is distinct from C++ references or reference semantics.

Options has its own allocation using optAlloc which allocates in a separate area, currently part of the options structure and utilizing "dynamic allocation" (dm_ functions on the Memory structure).

The Options structure are copied multiple times during a session, both as a backup (savedOptions) and into a separate options structure used by the Refactorer (refactoringOptions).

Since the options memory is then also copied, all pointers into the options memory need to be updated. To be able to do this, the options structure contains lists of addresses that needs to by "shifted".

When an option with a string or string list value is modified the option is registered in either the list of string valued options or the list of string list valued options. When an options structure is copied it must be performed using a deep copy function which "shifts" those options and their values (areas in the options memory) in the copy so that they point into the memory area of the copy, not the original.

After the deep copy the following point into the option memory of the copy

Arena allocators use bump pointer allocation:

This is extremely fast (O(1) allocation) but requires stack-like discipline for deallocation.

Arenas follow LIFO (last-in-first-out) cleanup:

Only the most recent allocation can be resized:

This constraint is enforced by guards in memory.c (see Development Environment chapter).

c-xrefactory uses specialized arenas for different purposes (see Modules chapter for details):

When an editor buffer is modified but not yet saved:

Without preload, the server would only see the last saved version of the file. The preload mechanism ensures that:

When a file is preloaded, the server must handle reference updates carefully:

The browser stack is a linked list of OlcxReferences entries, where each entry represents a symbol lookup session:

The browser stack is populated in two stages:

This separation means that browser stack entries can become stale if files are reparsed (e.g., with preloaded content) without refreshing the stack. Users typically need to pop and re-push to get fresh reference lists after edits.

Each BrowserMenu entry is a menu item wrapping a ReferenceableItem with UI presentation state:

Key insight: BrowserMenu is not just a menu - it’s a menu item. A collection of BrowserMenu items forms the actual menu shown to the user.

A single browser stack entry (OlcxReferences) can contain multiple BrowserMenu items:

This allows users to:

Browser menus are populated by scanning the referenceableItemTable:

Result: A persistent database mapping each entity (ReferenceableItem) to all its occurrences (References) across the entire codebase.

Result: Interactive navigation through the cross-reference database with selection and filtering.

This architecture separates concerns:

When currentInput runs out of lexems (read >= write), refillInputIfEmpty() uses the stream type to decide what to do:

Stream type must match buffer allocation:

Violating this invariant causes fatal errors when trying to free buffers from the wrong arena.

Pushing a NORMAL_STREAM onto the macro stack, then trying to free it as if it were MACRO_STREAM:

To implement incremental updates, c-xrefactory tracks when each file/buffer was last parsed.

C-xrefactory has two update strategies that trade off speed against completeness.

References flow through multiple stages in c-xrefactory, with different storage locations and ownership models at each stage.

The key insights:

C-xref started (probably) as a cross-referencer for the languages supported (C, Java, C++), orginally had the name "xref" which became "xrefactory" when refactoring support was added. And when Mariàn released a "C only" version in 2009 most of all the "xref" references and names was changed to "c-xref". So, as most software, there is a history and a naming legacy to remember.

Here are some of the conventions in naming that are being used:

The source code for c-xrefactory was using a very old C style with a separate proto.h where all prototypes for all externally visible functions were placed. Definitions are all over the place and it was hard to see where data is actually declared. This must change into module-oriented include-strategy.

Of course this will have to change into the modern x.h/x.c externally visible interface model so that we get clean modules that can be unittested.

The function prototypes have been now moved out to header files for each "module". Some of the types have also done that, but this is still a work in progress.

The preprocessor macro expansion code uses arena allocators (ppmMemory, macroBodyMemory) with stack-like discipline. Arena allocators are fast (pointer bumping) but require careful lifetime management.

There are not very many unittests at this point, only covering a quarter of the code. The "units" in this project are unclear and entangled so creating unittests is hard since it was not build to be tested, test driven or even clearly modularized.

All unittests use Cgreen as the unittest framework. If you are unfamiliar with it the most important point is that it can mock functions, so you will find mock implementations of all external functions for a module in a corresponding <module>.mock file.

Many modules are at least under test, meaning there is a <module>_tests.c in the unittest directory. Often only containing an empty test.

In the tests directory you will find tests that exercise the external behaviour of c-xref, "acceptance tests" or "system tests". Some tests actually do only that, they wouldn’t really count as tests as there are no verification except for the code being executed.

There are two basic strategies for the tests:

Some tests do not even test its output in any meaningful way and only provide coverage.

Some tests do a very bad job at verifying, either because my understanding at that time was very low, or because it is hard to verify the output. E.g. a "test" for generating references might only grepping the CXrefs files for some strings, not verifying that they actually point to the correct place.

Hopefully this will change as the code gets into a better state and the understanding grows.

Tests live in the tests directory and are auto-discovered by name: any directory starting with test_ will be recognized as a test case.

Each test typically includes:

Most tests use tests/Makefile.boilerplate which provides common macros:

The key macros are:

When $(VERIFY) passes, the output file is removed. This means you can easily identify failing tests by looking for test directories that still contain an output file. The utils/failing script lists these.

To suspend a test (skip it during test runs), create a .suspended file in the test directory.

Since all(?) c-xref operation rely on an options file which must contain absolute file paths (because the server runs as a separate process) it must be generated whenever the tests are to be run in a different location (new clone, test was renamed, …​).

This is performed by using a common template in tests and a target in tests/Maefile.boilerplate.

Each test should have a clean target that removes any temporary and generated files, including the .c-xrefrc file and generated references. This way it is easy to ensure that all tests have updated .c-xrefrc files.

Since many operations are performed from the editor, and the editor starts an "edit server" process, many tests need to emulate this behaviour.

The edit server session is mostly used for navigation. Refactorings are actually performed as separate invocations of c-xref.

In utils there is a server_driver.py script, which will take as input a file containing a sequence of commands. You can use this to start an edit, refactory or reference server session and then feed it with commands in the same fashion as an editor would do. The script also handles the communication through the buffer file (see [Editor Interface](./Design:-Editor-Interface)).

You can relatively easy re-create a sequence of interactions by using the sandboxed Emacs in tests/sandboxed_emacs.

There are two ways to use it, "make spy" or "make pure". With the "spy" an intermediate spy is injected between the editor and the edit server, capturing the interaction to a file.

With "pure" you just get the editor setup with c-xref-debug-mode and c-xref-debug-preserve-tmp-files on. This means that you can do what ever editor interactions you want and see the communication in the *Messages* buffer. See [Editor Interface](./Design:-Editor-Interface) for details.

Once you have figure out which part of the *Messages* buffer are interesting you can copy that out to a file and run utils/messages2commands.py on it to get a file formatted for input to server_driver.py.

utils/covers.py is a Python script that, in some enviroments, can list which test cases execute a particular line.

This is handy when you want to debug or step through a particular part of the code. Find a test that covers that particular line and run it using the debugger (usually make debug in the test directory).

Synopsis:

utils/sandboxed starts a sandboxed Emacs that uses the current elisp code and the c-xref from src. This allows you to run a test changes without having to polute your own setup.

This actually runs the tests/sandboxed_emacs pure version, which also sets up a completely isolated Emacs environment with its own packages loaded, configuration etc. See below.

Synopsis:

Problem: c-xrefactory was incorrectly expanding function-like macros during ## token pasting, violating C99 §6.10.3.3 which mandates that operands of ## should NOT be macro-expanded before concatenation.

Impact: Caused SIGSEGV in test_ffmpeg when processing complex standard library macros like INT64_MAX which expands to (INT64_C(9223372036854775807)) where INT64_C(c) is defined as c ## L.

Fix: Removed macro expansion logic from collate() function in src/yylex.c:1749-1797. Now matches GCC behavior exactly.

Test Updates: Updated 5 test cases to match correct GCC preprocessor behavior:

Problem: Large projects like ffmpeg were hitting the 30MB stackMemory limit during symbol table construction.

Root Cause: Stack memory is a persistent arena allocator that accumulates symbol tables across all parsed files. Growth is expected and normal for large projects.

Fix: Increased SIZE_stackMemory from 30MB to 100MB in src/constants.h:19.

Diagnostic Additions: Added nesting level and stack index tracking in src/xref.c to monitor memory usage patterns and detect unbalanced block nesting.

Decision: Removed the complex lexem stream caching mechanism to enable architectural refactoring.

Impact: Performance regression for large multi-file parsing (6.4× slowdown on test_ffmpeg/test_systemd), but enables critical modernization work.

Rationale: The ~300 lines of cache code were deeply intertwined with parsing and memory management, blocking refactoring efforts. Removal provides clearer code structure at the cost of temporary performance regression.

See: ADR-0012: Remove Lexem Stream Caching for detailed rationale, trade-offs, and future optimization strategies.

Current Status: 85%+ overall coverage achieved (January 2025)

Priority: MEDIUM (maintenance mode)

Ongoing Work:

Current Status: Experimental support for modern editor integration

Priority: HIGH - Opens c-xrefactory to wider audience beyond Emacs

Value: Enable VS Code, Neovim, and other LSP-capable editors to use c-xrefactory’s refactoring and navigation features.

Current Capabilities:

Key Limitations:

Next Steps: See Chapter 16: LSP Integration for details and Chapter 17 for technical architecture.

Current Status: Architecture documentation mostly complete

Priority: MEDIUM

Goals:

LexemStream API: Reduce pointer parameter proliferation in macro expansion code (6+ params → 3-4). Foundation for further modularization.

Macro Module Extraction: Separate 800 lines of macro expansion logic from yylex.c for better testability and maintainability.

Header Re-parsing Optimization: Address performance regression from cache removal. Make incremental builds faster.

Clean Persistence Store: Separate in-memory reference database from disk storage (.cx files). Enables LSP integration and testing without files.

Unified Symbol Database: Transition from batch-mode (-create required) to on-demand parsing. Enables zero cold-start for LSP.

Multi-file Support: Build include graph, enable cross-file navigation. Critical for full LSP feature parity.

Reorganize code by moving function definitions between source files while maintaining correctness. Essential for refactoring large codebases into better module structure.

Example: Extract a utility function from main.c to utils.c, automatically handling visibility changes and header declarations.

Smarter header placement: Automatically find the right location in header files based on existing declarations and dependencies.

Helper function detection: Identify and optionally move static helper functions that the moved function depends on. Prevents broken builds.

Include management: Automatically add necessary #include directives based on dependencies.

Preview: Show what will be changed before applying the refactoring.

For technical implementation details, see Chapter 17: Clean Parser API.

Language Server Protocol (LSP) integration enables c-xrefactory to work with modern editors and IDEs (VS Code, vim/neovim with LSP plugins, etc.) beyond Emacs. This opens c-xrefactory’s refactoring and navigation capabilities to a much wider audience.

Go to Definition (textDocument/definition):

File Parsing:

Single-file scope: Only symbols in the currently opened file are accessible. To navigate to a function in another file, you must first open that file.

No local variables: Go-to-definition doesn’t work for local variables or function parameters. This works in Emacs mode but requires architectural changes for LSP.

Missing LSP features: Only textDocument/definition is implemented. Coming features:

See the README for LSP client configuration examples. Basic setup:

The cxfile module currently mixes multiple responsibilities that should be separated:

This mixing creates tight coupling, makes testing difficult, and prevents future architectural improvements (like LSP integration or alternative storage backends).

The current code conflates two distinct concepts:

This refactoring is the foundation for the larger Unified Symbol Database Architecture (next section):

Migration path:

The current symbol database has evolved from a batch cross-referencer (like ctags) with artificial distinctions between "file-based" and "on-demand" modes. This creates:

The existing system distinguishes between:

The yylex.c file is 2353 lines and combines multiple responsibilities:

The macro expansion code is a substantial, cohesive subsystem that would benefit from extraction into its own module. Currently, it’s deeply embedded in yylex.c, making both lexing and macro expansion harder to understand and test in isolation.

The macro expansion system in yylex.c comprises:

Extract macro expansion into a new module: macroexpansion.c/h

Moving functions between C source files is a frequent manual refactoring task that requires:

This is error-prone and tedious, especially for functions with complex dependencies.

The Java-supporting version at /home/thoni/Utveckling/c-xref-java contains substantial move refactoring infrastructure:

Key implementations in c-xref-java/src/refactory.c:

Estimation Approach: We use relative complexity estimates (Story Points) with Phase 1 as the baseline (1×). This is more meaningful for hobby/OSS development than calendar time estimates. As phases complete, we can track velocity and refine future estimates.

||=== ||Phase |Value | ||Phase 1: Basic Move ||High - Eliminates manual cut/paste, handles 80% of cases | ||Phase 2: Include Management ||High - Prevents build breakage, major time saver | ||Phase 3: Static Helpers ||Medium - Nice to have, improves completeness | ||Phase 4: Polish ||Low - Quality of life improvements ||===

Phases 1-2 provide most value, independently testable Leverages existing Java refactoring code, reduces complexity significantly

The recent implementation of textDocument/definition for LSP created a clean, operation-based parser API in parsing.h/c that successfully decouples high-level operations from low-level parsing details. This API discovery presents an opportunity to gradually modernize legacy parsing infrastructure.

What works: * Single-file definition lookup via clean parsing.h interface * Operation-based configuration (ParserOperation enum) * No reliance on .cx files for immediate queries

What doesn’t work yet: * Multi-file definitions (symbol defined in different file than reference) * Cross-file type resolution * Include graph persistence

The LSP definition finding currently fails when a symbol’s definition is in a different file from the reference. This requires:

More broadly, the legacy codebase still relies on magic string command routing to drive parsing behavior:

This makes the codebase fragile, hard to understand, and difficult to extend with new operations.

The new parsing.h interface provides operation-based configuration (see src/parsing.h for details):

Key innovations: * Enum-based operations - Discoverable list (IDE autocomplete shows all options) * Configuration struct - Single point of setup, no scattered global flags * Operation predicates - Self-documenting behavior (needsReferenceAtCursor, allowsDuplicateReferences) * Type-safe - Compile-time checking replaces runtime magic string matching

New Parser API: * src/parsing.h - Public interface (167 lines) * src/parsing.c - Implementation with bridges (187 lines) * Functions: getParserOperation(), needsReferenceAtCursor(), parseToCreateReferences()

Legacy code currently using magic strings: * src/cxref.c - 20+ parseBufferUsingServer() calls * src/refactory.c - 10+ operations * src/extract.c - Multiple parsing passes * src/complete.c - Completion-specific parsing * src/server.c - Operation dispatch

Files for multi-file support: * src/reference_database.h/c - Add symbol type info * src/yylex.c - Build include graph * src/filetable.h/c - Persist include relationships

Phase 1: * ✓ New ParsingConfig API works alongside legacy strings * ✓ No performance regression * ✓ Both code paths tested and working

Phase 2: * ✓ 50%+ of parsing callsites use new API * ✓ Multi-file definition lookup works * ✓ All existing tests still pass

Phase 3 (if pursued): * ✓ All parsing driven by enum operations * ✓ Magic strings completely removed * ✓ New developers understand control flow

This refactoring supports: * ADR-0014 (On-Demand Parsing) - Single code path for all operations * Unified Symbol Database (next major refactoring) - Requires clean parsing API * LSP Multi-File Support - Immediately enables cross-file definition lookup

The key insight is that we can backport this gradually without breaking anything. The bridge pattern (new API calls old code for now) lets us improve incrementally, retiring magic strings and globals one section at a time.

Phase 1 proves the approach works with minimal risk. Phases 2-3 build on proven foundation, so failure to complete them doesn’t harm existing functionality.

Multi-file definitions are one motivating use case, but the real long-term benefit is a codebase where parsing behavior is discoverable and type-safe instead of driven by magic strings scattered throughout 25-year-old code.

Symbol navigation (PUSH/NEXT/PREVIOUS/POP) merges references from two sources:

When you PUSH on a symbol, the navigation menu creation (function createSelectionMenuForOperation) does:

This dual-source approach allows navigation without full project parse while providing updated positions for modified files.

The Emacs client only preloads the currently active file, not all modified editor buffers.

This creates incorrect navigation after editing non-current files:

We attempted to fix this (commit 8052518a4) by detecting modified files via timestamp comparison and reparsing them in processModifiedFilesForNavigation. This failed because:

As of 2025-12-22, the limitation remains documented but unfixed. The attempted fix was reverted to preserve the original working architecture.

MUST be maintained:

The table can NEVER reflect non-preloaded modified files because the server doesn’t receive that information from the client.

Implemented. TBD.

Implemented. TBD.

Implemented. TBD.

Purpose: Convert functions that are only used within their compilation unit to static storage class for better encapsulation and compiler optimization.

When to use:

Input:

Availability:

When cursor is on a non-static function definition where all callers are in the same file.

Algorithm:

Output:

Example:

Benefits:

Notes:

Input:

Algorithm:

Output:

Notes:

Purpose: Rename a file appearing in an include and update all the include directives

When to use:

Input:

Availability

When the cursor is on an #include directive. The file it references will be the "source".

Algorithm:

Output:

See Chapter 16a: Planned Refactoring Features for detailed design and implementation status.

Proposed refactoring to move a function definition from one C source file to another while automatically managing visibility (static vs extern) and potentially adding necessary declarations and includes.

Status: Phase 1 MVP complete.

Tentative.

Tentative.

Purpose: Convert functions that are only used within their compilation unit to static storage class for better encapsulation and compiler optimization.

When to use:

Input:

Availability:

When cursor is on a non-static function definition where all callers are in the same file.

Algorithm:

Output:

Example:

Benefits:

Notes:

Static memory (SM_ prefix) are static areas allocated by the compiler which is then indexed using a similarly named index variable (e.g. ftMemory and ftMemoryIndex), something the macros took advantage of. These are

One special case of static memory also exist:

These areas cannot be extended, when it overruns the program stops.

Cache Point Model: The system placed strategic snapshots of parser state at external definition boundaries (functions, global variables, etc.). When files were re-processed, the system could validate cache integrity, recover from cache points, and resume parsing only from the first changed definition onward.

Separation of Concerns: Recent refactoring had separated file tracking from cache validation:

Cache Point Management (caching.c):

File Modification Tracking:

The FileItem structure maintained multiple timestamp fields for tracking file modification:

Input Stream Caching:

C Parser Integration: Both C and Yacc parsers placed cache points after each external_definition, but only when not processing include files (includeStack.pointer == 0).

Parser-Specific Behavior:

The caching system was deeply integrated throughout the parsing pipeline:

Cache Hit Scenarios:

Optimization Benefits:

Once the FILL-macros was removed, we could move the enum-generation to use the actual c-xref. So from now on we build c-xref directly from the sources in the repo. Changes to any enums will trigger a re-generation of the enumTxt-files but since the enumTxt-files are only conversion of enum values to strings any mismatch will not prevent compilation, and it would even be possible to a manual update. This is a big improvement over the previous situation!

As indicated in FILL macros the bootstrapping of FILL-macros has finally and fully been removed.

Gone is also the compiler_defines.h, which was just removed without any obvious adverse effects. Maybe that will come back and bite me when we move to more platforms other than linux and MacOS…​

Left is, at this point, only the enumTxt generation, so most of the text below is kept for historical reasons.

c-xref uses a load of structures, and lists of them, that need to be created and initialized in a lot of places (such as the parsers). To make this somewhat manageable, c-xref itself parses the strucures and generates macros that can be used to fill them with one call.

c-xref is also bootstrapped into reading in a lot of predefined header files to get system definitions as "preloaded definitions".

Why this pre-loading was necessary, I don’t exactly know. It might be an optimization, or an idea that was born early and then just kept on and on. In any case it creates an extra complexity building and maintaining and to the structure of c-xref.

So this must be removed, see below.

The bootstrapping uses c-xref's own capability to parse C-code and parse those structures and spit out filling macros, and some other stuff.

This is done using options like `-task_regime_generate' which prints a lot of data structures on the standard output which is then fed into generated versions of strFill, strTdef(no longer exists) and enumTxt by the Makefile.

The process starts with building a c-xref.bs executable from checked in sources. This compile uses a BOOTSTRAP define that causes some header files to include pre-generated versions of the generated files (currently strFill.bs.h and enumTxt.bs.h) which should work in all environments.

After the c-xref.bs has been built, it is used to generate strFill and enumTxt which might include specific structures for the current environment.

HOWEVER: if FILL macros are used for structures which are different on some platforms, say a FILE structure, that FILL macro will have difference number of arguments, so I’m not sure how smart this "smart" generation technique actually is.

TODO: Investigate alternative approaches to this generate "regime", perhaps move to a "class"-oriented structure with initialization functions for each "class" instead of macros.

In options.h there are a number of definitions which somehow are sent to the compiler/preprocessor or used so that standard settings are the same as if a program will be compiled using the standard compiler on the platform. At this point I don’t know exactly how this conversion from C declarations to compile time definitions is done, maybe just entered as symbols in one of the many symboltables?

Typical examples include "__linux" but also on some platforms things like "fpos_t=long".

I’ve implemented a mechanism that uses "gcc -E -mD" to print out and catch all compiler defines in compiler_defines.h. This was necessary because of such definitions on Darwin which where not in the "pre-programmed" ones.

TODO?: As this is a more general approach it should possibly completely replace the "programmed" ones in options.c?

To be able to print the string values of enums the module generate.c (called when regime was RegimeGenerate) could also generate string arrays for all enums. By replacing that with some pre-processor magic for the few that was actually needed (mostly in log_trace() calls) we could do away with that whole "generate" functionality too.

(Last commit with enum generation intact is https://github.com/thoni56/c-xrefactory/commit/aafd7b1f813f2c17c684ea87ac87a0be31cdd4c4.)

For some cases the string representing the value of an Enum is needed. c-xref handles this using the "usual" 'parse code and generate' method. The module generate.c does this generation too.

Also in options.h some standard-like include paths are added, but there is a better attempt in getAndProcessGccOptions() which uses the compiler/preprocessor itself to figure out those paths.

TODO?: This is much better and should really be the only way, I think.

Since at bootstrap there must exist FILL-macros with the correct field names this strategy is an obstacle to cleaning up the code since every field is referenced in the FILL macros. When a field (in a structure which are filled using the FILL macro) changes name, this will make initial compilation impossible until the names of that field is also changed in the strFill.bs.h file.

One way to handle this is of course to use c-xrefactory itself and rename fields. This requires that the project settings also include a pass with BOOTSTRAP set, which it does.

I’ve started removing this step. In TODO.org I keep a hierarchical list of the actions to take (in a Mikado kind of style).

The basic strategy is to start with structures that no other structure depends on. Using the script utils/struct2dot.py you can generate a DOT graph that shows those dependencies.

Removal can be done in a couple of ways

The strTdef.h was generated using the option -typedefs as a part of the old -task_regime_generate strategy and generated typedef declarations for all types found in the parsed files.

I also think that you could actually merge the struct definition with the typedef so that strTdef.h would not be needed. But it seems that this design is because the structures in proto.h are not a directed graph, so loops makes that impossible. Instead the typedefs are included before the structs:

This is now ideomatically solved using the structs themselves: