elfutils/elfutils
Watch
5
Fork
You've already forked elfutils
 3
forked from elfutils/elfutils-mirror
Code Pull requests 1 Activity Actions

Add split DWARF (DWO/DWP) support to debuginfod #3

Open
pablogsal wants to merge 7 commits from pablogsal/elfutils:dwp-main into main
pull from: pablogsal/elfutils:dwp-main
merge into: elfutils:main
Conversation 8 Commits 7 Files changed 22 +1433 -58
pablogsal commented 2026-01-10 19:28:47 +00:00
Member
Copy link

This PR enables debuginfod to serve split DWARF files, solving a long-standing pain point for developers debugging binaries compiled with -gsplit-dwarf. When GCC uses this flag, the bulk of debug information gets stored in separate .dwo files or combined .dwp packages rather than in the executable itself. These files are referenced by a 64-bit DWO ID embedded in the skeleton compilation unit. Until now, if the .dwo files weren't present on the local filesystem: a common situation when debugging binaries from package repositories: debuggers simply couldn't access the full debug information.

This PR enables debuginfod to serve split DWARF files, solving a long-standing pain point for developers debugging binaries compiled with -gsplit-dwarf. When GCC uses this flag, the bulk of debug information gets stored in separate .dwo files or combined .dwp packages rather than in the executable itself. These files are referenced by a 64-bit DWO ID embedded in the skeleton compilation unit. Until now, if the .dwo files weren't present on the local filesystem: a common situation when debugging binaries from package repositories: debuggers simply couldn't access the full debug information.
When GCC compiles with -gsplit-dwarf, debug information is stored in
separate .dwo files or combined .dwp packages, referenced by a 64-bit
DWO ID in the skeleton compilation unit. The libdw library currently
searches for these files only in local filesystem paths derived from
DW_AT_comp_dir and DW_AT_GNU_dwo_name. This fails when the .dwo files
are not present locally, such as when debugging a binary downloaded
from a package repository without its corresponding debug split files.

This introduces dwarf_set_dwo_lookup(), a public API that allows
applications to register a callback for custom DWO resolution. When
libdw cannot find a split unit locally, it invokes this callback with
the 64-bit DWO ID, expecting a file descriptor in return. The callback
approach was chosen over a direct debuginfod dependency to keep libdw
independent of network libraries and to allow applications like GDB to
implement their own resolution strategies. The callback receives an
opaque user_data pointer to support integration with application state
such as libdwfl module handles.

	* libdw/dwarf_set_dwo_lookup.c: New file implementing
	dwarf_set_dwo_lookup() which registers a callback for DWO
	resolution by 64-bit DWO ID.
	* libdw/libdw.h (dwarf_set_dwo_lookup): Declare new function.
	* libdw/libdwP.h (struct Dwarf): Add dwo_lookup_cb and
	dwo_lookup_user_data fields for callback storage.
	(INTDECL): Add dwarf_set_dwo_lookup.
	* libdw/libdw.map (ELFUTILS_0.193): Add dwarf_set_dwo_lookup.
	* libdw/Makefile.am (libdw_a_SOURCES): Add dwarf_set_dwo_lookup.c.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
The existing split unit search in __libdw_find_split_unit() tries
several local paths based on DW_AT_GNU_dwo_name and DW_AT_comp_dir,
but has no fallback when files are not present. Debuggers working with
binaries from package repositories often lack the corresponding .dwo
files on the local filesystem, leaving split DWARF units unresolvable.

This refactors try_split_file() to separate the file descriptor logic
into try_split_fd(), enabling reuse with file descriptors obtained from
sources other than local paths. After exhausting local search paths,
__libdw_find_split_unit() now invokes the callback registered via
dwarf_set_dwo_lookup() if one exists. The callback pointer and user
data are read under the dwarf_lock mutex to ensure thread-safe access,
matching the synchronization in dwarf_set_dwo_lookup(). This also fixes
a minor memory leak in try_dwp_file() where dwp_dwarf was not freed
when the DWP file was already open.

	* libdw/libdw_find_split_unit.c (try_split_fd): New function
	extracted from try_split_file(), takes ownership of fd.
	(try_split_file): Refactored to call try_split_fd.
	(try_dwp_file): Fix memory leak by calling dwarf_end on
	dwp_dwarf when DWP already open.
	(__libdw_find_split_unit): Invoke dwo_lookup_cb callback when
	local search fails, reading callback under dwarf_lock mutex.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
Build systems using -gsplit-dwarf produce .dwo files and .dwp packages
that contain the bulk of debug information, referenced by 64-bit DWO
IDs from skeleton compilation units. These files could not be served
by debuginfod because the server only indexed content by build-id, and
DWO/DWP files typically lack build-ids entirely since they are not
linked into the final executable.

This extends the database schema to buildids11, adding tables to track
DWO content: _dwoids for interning 64-bit IDs as 16-char hex strings,
_f_dwo for file-based DWO content, and _r_dwo for archive-based content.
The schema mirrors the existing _f_de/_r_de pattern for consistency.
During scanning, dwarf_extract_dwo_ids() identifies files containing
DW_UT_split_compile or DW_UT_split_type units and extracts their IDs,
distinguishing .dwp packages by the presence of .debug_cu_index. The
elf_classify() function was modified to process files without build-ids
since this is normal for DWO content. A new /dwoid/{id}/debuginfo
endpoint queries the _query_dwo view and returns matching files, using
the same handle_buildid_match() infrastructure for cache-friendly
responses. Federation is not yet supported for dwoid queries as it
would require extending the client protocol.

	* debuginfod/debuginfod.cxx (BUILDIDS): Bump to buildids11 for
	schema change.
	(DEBUGINFOD_SQLITE_DDL): Add _dwoids table for DWO ID interning,
	_f_dwo and _r_dwo tables for file and archive DWO content with
	is_dwp flag, _query_dwo view for unified queries, update _stats
	view with DWO counts.  Add DROP statements for buildids10 tables.
	(handle_dwoid): New function to handle /dwoid/{id}/debuginfo
	endpoint, validates 16-char hex ID and queries _query_dwo view.
	(handler_cb): Add /dwoid URL routing with same blocking logic as
	/buildid endpoint.
	(dwarf_extract_dwo_ids): New function to extract DWO IDs from
	split DWARF files, detecting DWP files via .debug_cu_index.
	(elf_classify): Add dwo_ids and is_dwp output parameters, call
	dwarf_extract_dwo_ids, handle files without build-ids.
	(scan_source_file): Pass dwo_ids to elf_classify, insert into
	_dwoids and _f_dwo/_r_dwo tables.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
Applications like GDB and libdwfl need to fetch split DWARF files from
debuginfod servers, but the existing client API only supports queries
by build-id. DWO files are identified by their 64-bit DWO ID from the
skeleton compilation unit, not by build-id, so a different query
mechanism is required.

This adds debuginfod_find_dwo() which queries servers at the
/dwoid/{id}/debuginfo endpoint. The function accepts DWO IDs either as
a hex string (when dwo_id_len is 0) or as binary bytes in little-endian
order matching how libdw stores them internally. Binary input is
reversed to produce the canonical big-endian hex representation that
the server expects. A convenience wrapper debuginfod_find_dwo_by_id()
takes the 64-bit ID directly for simpler integration. The cache uses
a dwoid/ subdirectory parallel to the existing buildid/ structure.
The debuginfod-find command gains a "dwoid" subcommand for manual
testing and shell script integration. Both functions are exported in
the ELFUTILS_0.193 symbol version.

	* debuginfod/debuginfod.h.in: Include stdint.h for uint64_t.
	(debuginfod_find_dwo): Declare new function taking DWO ID as
	hex string or binary bytes.
	(debuginfod_find_dwo_by_id): Declare convenience wrapper taking
	uint64_t DWO ID directly.
	* debuginfod/debuginfod-client.c (debuginfod_find_dwo): New
	function implementing DWO lookup via /dwoid endpoint, handles
	hex string and binary input, caches in dwoid/ subdirectory.
	(debuginfod_find_dwo_by_id): New wrapper converting uint64_t
	to little-endian bytes and calling debuginfod_find_dwo.
	* debuginfod/debuginfod-find.c (args_doc): Add dwoid usage.
	(main): Handle "dwoid" subcommand calling debuginfod_find_dwo.
	* debuginfod/libdebuginfod.map (ELFUTILS_0.193): Add
	debuginfod_find_dwo and debuginfod_find_dwo_by_id symbols.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
When libdwfl opens a module containing split DWARF, the skeleton
compilation units reference .dwo files by DWO ID. Previously, libdw
would only search local filesystem paths, leaving split units
unresolved when the corresponding .dwo files were not present. Users
debugging binaries from package repositories had to manually obtain
and place .dwo files in the expected locations.

This registers a DWO lookup callback with libdw when loading DWARF
data for a module. The callback invokes __libdwfl_debuginfod_find_dwo()
which uses the debuginfod_find_dwo_by_id() client function to query
configured debuginfod servers. The callback receives the Dwfl handle
as user_data to access the shared debuginfod client with its progress
callbacks and URL configuration. The debuginfod_find_dwo_by_id symbol
is loaded via dlsym and treated as optional to maintain compatibility
with older libdebuginfod versions. This integration is conditional on
ENABLE_LIBDEBUGINFOD, preserving builds without debuginfod support.

	* libdwfl/libdwflP.h (__libdwfl_debuginfod_find_dwo): Declare
	new internal function for DWO lookup via debuginfod.
	* libdwfl/debuginfod-client.c (fp_debuginfod_find_dwo_by_id):
	New static function pointer for dlsym lookup.
	(__libdwfl_debuginfod_find_dwo): New function calling
	debuginfod_find_dwo_by_id via function pointer.
	(__libdwfl_debuginfod_init): Load debuginfod_find_dwo_by_id
	symbol, treat as optional for compatibility.
	* libdwfl/dwfl_module_getdwarf.c (dwfl_dwo_lookup_callback):
	New static callback function invoking debuginfod DWO lookup.
	(load_dw): Register dwfl_dwo_lookup_callback with libdw via
	dwarf_set_dwo_lookup when ENABLE_LIBDEBUGINFOD defined.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
The new DWO lookup functionality needs end-to-end testing covering
the server indexing, client queries, and libdwfl integration. Without
tests, regressions in split DWARF handling could go unnoticed since
this code path only activates when .dwo files are missing locally but
available via debuginfod.

This adds run-debuginfod-dwoid.sh which tests the complete DWO lookup
flow. It starts a debuginfod server indexing .dwp and .dwo test files,
verifies the server correctly extracts DWO IDs during scanning, tests
debuginfod-find dwoid queries, and validates that libdwfl can resolve
skeleton compilation units via the debuginfod callback. The test uses
DWARF 4 and DWARF 5 fixtures to cover both split DWARF generations.
A helper program debuginfod_dwoid_find.c exercises the libdwfl path
by loading skeleton files and verifying split units are resolved. The
run-debuginfod-seekable.sh test is updated to reference the new
buildids11 schema version for the seekable table drop test.

	* tests/debuginfod_dwoid_find.c: New test program that loads a
	skeleton file via libdwfl and verifies split DWARF units are
	resolved through the debuginfod callback.
	* tests/run-debuginfod-dwoid.sh: New test script exercising DWO
	lookup via debuginfod server, client command, and libdwfl.
	* tests/testfile-dwp-4.dwp: New DWARF 4 DWP test fixture.
	* tests/testfile-dwp-5: New DWARF 5 skeleton file test fixture.
	* tests/testfile-dwp-5.dwp: New DWARF 5 DWP test fixture.
	* tests/Makefile.am (check_PROGRAMS): Add debuginfod_dwoid_find.
	(TESTS): Add run-debuginfod-dwoid.sh.
	(EXTRA_DIST): Add run-debuginfod-dwoid.sh.
	(debuginfod_dwoid_find_LDADD): Define link libraries.
	* tests/run-debuginfod-seekable.sh: Update table name from
	buildids10_r_seekable to buildids11_r_seekable.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
pablogsal commented 2026-01-10 19:29:23 +00:00
Author
Member
Copy link

Tested locally:

$ make check -j 

...

============================================================================
Testsuite summary for elfutils 0.194
============================================================================
# TOTAL: 293
# PASS:  285
# SKIP:  8
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
Tested locally: ``` $ make check -j ... ============================================================================ Testsuite summary for elfutils 0.194 ============================================================================ # TOTAL: 293 # PASS: 285 # SKIP: 8 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ```
pablogsal commented 2026-01-10 19:39:42 +00:00
Author
Member
Copy link

Specific run for the new tests:

$ make check TESTS=run-debuginfod-dwoid.sh -C tests

PASS: run-debuginfod-dwoid.sh
============================================================================
Testsuite summary for elfutils 0.194
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
Specific run for the new tests: ``` $ make check TESTS=run-debuginfod-dwoid.sh -C tests PASS: run-debuginfod-dwoid.sh ============================================================================ Testsuite summary for elfutils 0.194 ============================================================================ # TOTAL: 1 # PASS: 1 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ```
Running make check generates test binaries, log files, and intermediate
build outputs that were not properly ignored. These files would show up
as untracked in git status, cluttering the working directory view.

This adds the missing patterns to the existing gitignore files and
creates a new one for libelf. The patterns follow the same style used
elsewhere in the project, ignoring test binaries by name and using
wildcards for log and trs files.

	* libdw/.gitignore: Add *.log, *.trs, and dwarf_srclang_check.
	* libelf/.gitignore: New file. Add *.log, *.trs, and
	gelf_fsize_check.
	* po/.gitignore: Add insert-header.sed.
	* tests/.gitignore: Add debuginfod_dwoid_find.

Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
mjw requested review from mjw 2026-01-10 20:10:05 +00:00
pablogsal commented 2026-01-10 21:36:41 +00:00
Author
Member
Copy link

I tested also a patched gdb with a server from this PR:

binutils-gdb master  ? ❯ export LD_LIBRARY_PATH=/tmp/elfutils-install/lib:$LD_LIBRARY_PATH

binutils-gdb master  ? ❯ export DEBUGINFOD_URLS="http://localhost:8002"

binutils-gdb master  ? ❯ ./build/gdb/gdb \
        --data-directory=/home/pablogsal/github/binutils-gdb/build/gdb/data-directory \
        --batch \
        -eiex "set debuginfod enabled on" \
        -ex "echo === Setting breakpoint on main ===\n" \
        -ex "break main" \
        -ex "echo \n=== Running program ===\n" \
        -ex "run" \
        -ex "echo \n=== Source listing (proves DWO was loaded) ===\n" \
        -ex "list" \
        -ex "echo \n=== Local variables ===\n" \
        -ex "info locals" \
        -ex "echo \n=== Step into add() function ===\n" \
        -ex "step" \
        -ex "step" \
        -ex "echo \n=== Now inside add() - showing args ===\n" \
        -ex "info args" \
        -ex "echo \n=== Backtrace ===\n" \
        -ex "backtrace" \
        /tmp/test
=== Setting breakpoint on main ===
debuginfod_find_dwo 721cb9e895e0533b
server urls "http://localhost:8002"
init server 0 http://localhost:8002/dwoid
url 0 http://localhost:8002/dwoid/3b53e095e8b91c72/debuginfo
query 1 urls in parallel
header HTTP/1.1 200 OK
header Date: Sat, 10 Jan 2026 21:36:32 GMT
header Content-Type: application/octet-stream
header X-DEBUGINFOD-SIZE: 1552
header X-DEBUGINFOD-FILE: /tmp/test.dwo.backup
header Last-Modified: Sat, 10 Jan 2026 20:03:09 GMT
header Cache-Control: public
header Content-Length: 1552

X-DEBUGINFOD-SIZE: 1552
X-DEBUGINFOD-FILE: /tmp/test.dwo.backup
committed to url 0
Breakpoint 1 at 0x115b: file /tmp/test.c, line 9.


...

Breakpoint 1, main () at /tmp/test.c:9
9           int x = 42;

=== Source listing (proves DWO was loaded) ===
4           int result = a + b;
5           return result;
6       }
7
8       int main() {
9           int x = 42;
10          int y = 13;
11          int sum = add(x, y);
12          printf("Hello, x = %d, y = %d, sum = %d\n", x, y, sum);
13          return 0;

=== Local variables ===
x = 0
y = -134340976
sum = 32767

=== Step into add() function ===
10          int y = 13;
11          int sum = add(x, y);

=== Now inside add() - showing args ===
No arguments.

=== Backtrace ===
#0  main () at /tmp/test.c:11
I tested also a patched gdb with a server from this PR: ``` binutils-gdb master  ? ❯ export LD_LIBRARY_PATH=/tmp/elfutils-install/lib:$LD_LIBRARY_PATH binutils-gdb master  ? ❯ export DEBUGINFOD_URLS="http://localhost:8002" binutils-gdb master  ? ❯ ./build/gdb/gdb \ --data-directory=/home/pablogsal/github/binutils-gdb/build/gdb/data-directory \ --batch \ -eiex "set debuginfod enabled on" \ -ex "echo === Setting breakpoint on main ===\n" \ -ex "break main" \ -ex "echo \n=== Running program ===\n" \ -ex "run" \ -ex "echo \n=== Source listing (proves DWO was loaded) ===\n" \ -ex "list" \ -ex "echo \n=== Local variables ===\n" \ -ex "info locals" \ -ex "echo \n=== Step into add() function ===\n" \ -ex "step" \ -ex "step" \ -ex "echo \n=== Now inside add() - showing args ===\n" \ -ex "info args" \ -ex "echo \n=== Backtrace ===\n" \ -ex "backtrace" \ /tmp/test === Setting breakpoint on main === debuginfod_find_dwo 721cb9e895e0533b server urls "http://localhost:8002" init server 0 http://localhost:8002/dwoid url 0 http://localhost:8002/dwoid/3b53e095e8b91c72/debuginfo query 1 urls in parallel header HTTP/1.1 200 OK header Date: Sat, 10 Jan 2026 21:36:32 GMT header Content-Type: application/octet-stream header X-DEBUGINFOD-SIZE: 1552 header X-DEBUGINFOD-FILE: /tmp/test.dwo.backup header Last-Modified: Sat, 10 Jan 2026 20:03:09 GMT header Cache-Control: public header Content-Length: 1552 X-DEBUGINFOD-SIZE: 1552 X-DEBUGINFOD-FILE: /tmp/test.dwo.backup committed to url 0 Breakpoint 1 at 0x115b: file /tmp/test.c, line 9. ... Breakpoint 1, main () at /tmp/test.c:9 9 int x = 42; === Source listing (proves DWO was loaded) === 4 int result = a + b; 5 return result; 6 } 7 8 int main() { 9 int x = 42; 10 int y = 13; 11 int sum = add(x, y); 12 printf("Hello, x = %d, y = %d, sum = %d\n", x, y, sum); 13 return 0; === Local variables === x = 0 y = -134340976 sum = 32767 === Step into add() function === 10 int y = 13; 11 int sum = add(x, y); === Now inside add() - showing args === No arguments. === Backtrace === #0 main () at /tmp/test.c:11 ```
mjw commented 2026-01-13 22:16:51 +00:00
Owner
Copy link

Could you hit the "Update branch by rebase" button (it is under the Update branch by merge button)? The forgejo/workflow files weren't in place yet, so no Actions were triggered. I would like to see how/if that works after a rebase. It might not work, I am new to forgejo. But it would be fun to see if it does.

Could you hit the "Update branch by rebase" button (it is under the Update branch by merge button)? The forgejo/workflow files weren't in place yet, so no Actions were triggered. I would like to see how/if that works after a rebase. It might not work, I am new to forgejo. But it would be fun to see if it does.
pablogsal force-pushed dwp-main from 294c299608 to ea8dce12fb
All checks were successful
fedora-build-tests / check (pull_request) Successful in 5m47s
Details
debian-build-tests / check (pull_request) Successful in 7m12s
Details
2026-01-14 01:56:00 +00:00 Compare
pablogsal commented 2026-01-14 01:57:21 +00:00
Author
Member
Copy link

I clicked the button and saw a message saying:

Merge conflict checking is in progress. Try again in few moments.

Then I saw:

pablogsal force-pushed dwp-main from 294c299608 to ea8dce12fb now

I clicked the button and saw a message saying: > Merge conflict checking is in progress. Try again in few moments. Then I saw: > pablogsal force-pushed dwp-main from 294c299608 to ea8dce12fb now
mjw commented 2026-01-14 13:08:58 +00:00
Owner
Copy link

Hohum, that didn't work :{
Let me try closing and reopening...
Sorry for "playing" with this instead of properly reviewing it.

Hohum, that didn't work :{ Let me try closing and reopening... Sorry for "playing" with this instead of properly reviewing it.
mjw commented 2026-01-14 13:09:58 +00:00
Owner
Copy link

Cool that worked! There is now " Some checks are pending".

Cool that worked! There is now " Some checks are pending".
mjw commented 2026-01-14 13:13:08 +00:00
Owner
Copy link

I had to explicitly Approve the checks. But I guess that is fair. Just means you need some kind of sponsor to run Actions on merge requests from forks.

I had to explicitly Approve the checks. But I guess that is fair. Just means you need some kind of sponsor to run Actions on merge requests from forks.
All checks were successful
fedora-build-tests / check (pull_request) Successful in 5m47s
Details
debian-build-tests / check (pull_request) Successful in 7m12s
Details
This pull request has changes conflicting with the target branch.
  • debuginfod/debuginfod-find.c
  • debuginfod/debuginfod.h.in
  • debuginfod/libdebuginfod.map
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u dwp-main:pablogsal-dwp-main
git switch pablogsal-dwp-main

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff pablogsal-dwp-main
git switch pablogsal-dwp-main
git rebase main
git switch main
git merge --ff-only pablogsal-dwp-main
git switch pablogsal-dwp-main
git rebase main
git switch main
git merge --no-ff pablogsal-dwp-main
git switch main
git merge --squash pablogsal-dwp-main
git switch main
git merge --ff-only pablogsal-dwp-main
git switch main
git merge pablogsal-dwp-main
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
mjw
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
elfutils/elfutils!3
No description provided.