How do I get a stacktrace with gdb and install debug symbols for GNOME Remote Desktop?

jno · June 29, 2024, 4:09am

I’m experiencing an issue with GNOME Remote Desktop on Fedora, and I need to generate a stacktrace:

What you could try is attaching gdb in these situations (when having debug symbols installed), and then attach gdb to g-r-d, get a stacktrace, restart it, and look where things would block.

I installed gdb but could not figure out how to install debug symbols for g-r-d. Fedora doesn’t seem to have a package for it.

I tried to attach gdb to the running g-r-d process (/usr/libexec/gnome-remote-desktop-daemon --system) with gdb -p 79564. I enabled debuginfod and it downloaded a lot of debug symbols, but I don’t know if any of them were relevant:

Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007eff97d1d75d in __GI___poll (fds=0x564a89d30fe0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29        return SYSCALL_CANCEL (poll, fds, nfds, timeout)

When I tried to connect to g-r-d from Remmina and typed c to continue in gdb, all I got was this:

[New Thread 0x7eff8b4006c0 (LWP 98160)]
[New Thread 0x7eff8aa006c0 (LWP 98161)]
[New Thread 0x7eff90c006c0 (LWP 98162)]
[New Thread 0x7eff920006c0 (LWP 98163)]
[New Thread 0x7eff916006c0 (LWP 98164)]
[New Thread 0x7eff8be006c0 (LWP 98165)]
[New Thread 0x7eff8a0006c0 (LWP 98166)]
[New Thread 0x7eff896006c0 (LWP 98167)]
[New Thread 0x7eff88c006c0 (LWP 98168)]
[New Thread 0x7eff7fe006c0 (LWP 98169)]
[Thread 0x7eff88c006c0 (LWP 98168) exited]
[New Thread 0x7eff7f4006c0 (LWP 98170)]
[New Thread 0x7eff7ea006c0 (LWP 98171)]
[New Thread 0x7eff7e0006c0 (LWP 98172)]
[New Thread 0x7eff7d6006c0 (LWP 98173)]
[New Thread 0x7eff7cc006c0 (LWP 98174)]
[New Thread 0x7eff73e006c0 (LWP 98175)]
[New Thread 0x7eff734006c0 (LWP 98176)]
[New Thread 0x7eff72a006c0 (LWP 98177)]
[New Thread 0x7eff720006c0 (LWP 98178)]
[New Thread 0x7eff716006c0 (LWP 98179)]
[New Thread 0x7eff70c006c0 (LWP 98180)]
[Thread 0x7eff920006c0 (LWP 98163) exited]
[Thread 0x7eff896006c0 (LWP 98167) exited]
[Thread 0x7eff8a0006c0 (LWP 98166) exited]
[Thread 0x7eff8be006c0 (LWP 98165) exited]
[Thread 0x7eff916006c0 (LWP 98164) exited]
[Thread 0x7eff90c006c0 (LWP 98162) exited]
[Thread 0x7eff8aa006c0 (LWP 98161) exited]
[Thread 0x7eff7cc006c0 (LWP 98174) exited]
[Thread 0x7eff7f4006c0 (LWP 98170) exited]
[Thread 0x7eff7d6006c0 (LWP 98173) exited]
[Thread 0x7eff7e0006c0 (LWP 98172) exited]
[Thread 0x7eff7ea006c0 (LWP 98171) exited]
[Thread 0x7eff7fe006c0 (LWP 98169) exited]

My understanding is that there are supposed to be function calls here instead of Thread lines (AFAIU this is ASM-level debugging)? I’m assuming that’s due to the debug symbols for g-r-d not being installed on the server. But I don’t know how to get them on there.

The issue is intermittent and probably won’t happen again for a few days, but I want to prepare for the next time it happens.

I used these sources to try to figure out gdb but haven’t had much luck:

Neui · June 29, 2024, 11:49am

You are supposed to submit c when it asks you about something about paging (--Type <RET> for more, q to quit, c to continue without paging--, happens when gdb prints pages of lines). It didn’t occur, so you can ignore that step. As a command, c stands for continue (continue executing the program), which is what you have done. You can press CTRL+C in gdb to pause again and then execute thread apply all bt full (or whatever you need). It’ll probably ask you about the paging thing when you print the stacktrace (because they tend do be long), so you can use c there when it asks you.

mcatanzaro · June 29, 2024, 1:22pm

Neui’s advice is correct, but I’ll add that if you’re trying to get a stack trace for a crash, then your problem is: it is not crashing. Normally you want to get the stack trace at the point of the crash.

Pressing Ctrl+C to take a stack trace is generally only useful if you’re trying to debug a hang or an excessive CPU usage issue.

jno · June 29, 2024, 2:38pm

Thanks for the advice!

It’s not a crash. The linked issue is here: GNOME Remote Desktop stops accepting connections after a few hours, needs to be restarted (#213) · Issues · GNOME / gnome-remote-desktop · GitLab

GNOME Remote Desktop stops letting me connect to it after a random period of time. I can RDP to Windows servers elsewhere at the same time, but not g-r-d. When I restart the service, I can connect again.

So I guess it’s similar to a hang?

It’s still working fine for now. But I’m testing it out using these steps:

Turn Debugging on in Remmina
Connect to the g-r-d server
Run gdb -p 79564 on the server and enable debuginfod

But at this point, I just wait endlessly. Usually, Remmina would have connected or stopped trying by this point, but 5 minutes later it’s still trying to connect. The only way I could get it to connect was to submit c (Continue) and then the New Threads were created and it connects.

Otherwise the DEBUG output in Remmina just keeps looping:

(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0: owner-change event received
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is different than me: new=(nil) me=0x5e7a230bcdb0
(DEBUG) - (remmina_rdp_event_on_clipboard) - gp=0x5e7a230bcdb0 owner-change: new owner is not me: Sending local clipboard format list to server.
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) - gp=0x5e7a230bcdb0 sending to server the following local clipboard content formats
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain;charset=utf-8 will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format UTF8_STRING will be sent to remote as 13
(DEBUG) - (remmina_rdp_cliprdr_get_client_format_list) -      local clipboard format text/plain will be sent to remote as 1

I’m not sure what I need. But this does actually give me some function output! Which seems useful!

The last time I tried to run gdb on it while it was failing to connect, it seemed to actually fix the issue before I could even figure out how to get a stacktrace.

I’ll try this out next time I face the issue. Thanks!

Neui · June 29, 2024, 3:41pm

I somehow thought you attached GDB while it was in that hanging state and you did c (for the pager) because of the instructions on your first link (continuing a hanging program doesn’t seem to be that useful).

If you have the (gdb) prompt, then the program is paused (in general), so for external software it looks like it doesn’t answer (and may time out) until you c to continue.

The thread created/destroyed messages are printed by gdb (but the actions are made by the application), but in general they aren’t useful. You can ignore those messages. It just means the program is running and does something.

CTRL+C in GDB while running pauses the program and gives you a (gdb) prompt back, so when it hangs you can press CTRL+C and execute thread apply all bt full to get the stack trace for all threads (bt full alone just prints for one thread, which may not be useful). You can continue later if you want, or quit to detach.