Startup Leak Issue noticed when selecting Wayland as the `GLFW` platform

[jamylak]

To reproduce

You can use nix develop or just follow install instructions on README

cmake -B build -DCMAKE_BUILD_TYPE=Debug
./build/vsdf --toy example.frag --frames 1

on Debug Build running on KDE Wayland running on Nix in VM on Macbook M1

And Sanitizer finds a leak

I didnt see this issue show on Mac or Windows I think??

Indirect leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0xffff8f474358 in strdup (/nix/store/qipd93x9gjyiygqk673rd2ssnf8y7jj0-gcc-14.3.0-lib/lib/libasan.so.8+0xd4358)
    #1 0xffff882971c4 in IA__FcValueSave (/nix/store/a5bwqrnnn86siynb6nc7qwbhliwvqpzx-fontconfig-2.17.1-lib/lib/libfontconfig.so.1+0x271c4)
    #2 0xffff882985bc in FcPatternObjectAddWithBinding (/nix/store/a5bwqrnnn86siynb6nc7qwbhliwvqpzx-fontconfig-2.17.1-lib/lib/libfontconfig.so.1+0x285bc)
    #3 0xffff84d1ffd8 in pango_fc_font_map_load_fontset (/nix/store/k9vwsjba5biwz9z4vskcsayjjhfx94vj-pango-1.57.0/lib/libpangoft2-1.0.so.0+0xffd8)
    #4 0xffff84e8eb24 in pango_context_get_metrics (/nix/store/k9vwsjba5biwz9z4vskcsayjjhfx94vj-pango-1.57.0/lib/libpango-1.0.so.0+0x2eb24)
    #5 0xffff82ba4830 in gtk_label_measure (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x224830) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #6 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #7 0xffff82c6fe08 in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2efe08) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #8 0xffff82a67d1c in gtk_box_get_content_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0xe7d1c) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #9 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #10 0xffff82c6fe08 in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2efe08) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #11 0xffff82b76148 in gtk_header_bar_get_content_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x1f6148) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #12 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #13 0xffff82c6fdb8 in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2efdb8) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #14 0xffff831478ec in draw_title_bar (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x78ec)
    #15 0xffff83147f28 in draw_decoration (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x7f28)
    #16 0xffff83148190 in libdecor_plugin_gtk_frame_commit (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x8190)
    #17 0xffff86535708 in libdecor_frame_commit (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor-0.so.0+0x5708)
    #18 0xffff8f32fa90 in createShellObjects (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x2fa90)
    #19 0xffff8f316378 in glfwCreateWindow (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x16378)
    #20 0xaaaad46971a8 in createGLFWwindow /home/james/vsdf-present-fix/include/glfwutils.h:39
    #21 0xaaaad46971a8 in OnlineSDFRenderer::glfwSetup() /home/james/vsdf-present-fix/src/online_sdf_renderer.cpp:43
    #22 0xaaaad46ab790 in OnlineSDFRenderer::setup() /home/james/vsdf-present-fix/src/online_sdf_renderer.cpp:26
    #23 0xaaaad467e588 in run(int, char**) /home/james/vsdf-present-fix/src/main.cpp:370
    #24 0xaaaad46747ac in main /home/james/vsdf-present-fix/src/main.cpp:385
    #25 0xffff8b2b62b0 in __libc_start_call_main (/nix/store/89n0gcl1yjp37ycca45rn50h7lms5p6f-glibc-2.40-66/lib/libc.so.6+0x262b0) (BuildId: 70fbec6c13b6ae06e6b049bcbad20e1b16b0694e)
    #26 0xffff8b2b6394 in __libc_start_main@GLIBC_2.17 (/nix/store/89n0gcl1yjp37ycca45rn50h7lms5p6f-glibc-2.40-66/lib/libc.so.6+0x26394) (BuildId: 70fbec6c13b6ae06e6b049bcbad20e1b16b0694e)
    #27 0xaaaad4674fac in _start (/home/james/vsdf-present-fix/build/vsdf+0x34fac)

Indirect leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0xffff8f474358 in strdup (/nix/store/qipd93x9gjyiygqk673rd2ssnf8y7jj0-gcc-14.3.0-lib/lib/libasan.so.8+0xd4358)
    #1 0xffff882971c4 in IA__FcValueSave (/nix/store/a5bwqrnnn86siynb6nc7qwbhliwvqpzx-fontconfig-2.17.1-lib/lib/libfontconfig.so.1+0x271c4)
    #2 0xffff882985bc in FcPatternObjectAddWithBinding (/nix/store/a5bwqrnnn86siynb6nc7qwbhliwvqpzx-fontconfig-2.17.1-lib/lib/libfontconfig.so.1+0x285bc)
    #3 0xffff84d1ffd8 in pango_fc_font_map_load_fontset (/nix/store/k9vwsjba5biwz9z4vskcsayjjhfx94vj-pango-1.57.0/lib/libpangoft2-1.0.so.0+0xffd8)
    #4 0xffff84e8eb24 in pango_context_get_metrics (/nix/store/k9vwsjba5biwz9z4vskcsayjjhfx94vj-pango-1.57.0/lib/libpango-1.0.so.0+0x2eb24)
    #5 0xffff82ba4830 in gtk_label_measure (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x224830) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #6 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #7 0xffff82c6f8cc in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2ef8cc) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #8 0xffff82a67bd4 in gtk_box_get_content_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0xe7bd4) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #9 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #10 0xffff82c6f8cc in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2ef8cc) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #11 0xffff82b7606c in gtk_header_bar_get_content_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x1f606c) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #12 0xffff82ace714 in gtk_css_gadget_get_preferred_size (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x14e714) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #13 0xffff82c6f8cc in gtk_widget_query_size_for_orientation (/nix/store/w9li4sk4g2ijslh3nbyl60mbb1rfjgc8-gtk+3-3.24.51/lib/libgtk-3.so.0+0x2ef8cc) (BuildId: c7e74bb4e748c7d676b41c893332c36183288a5f)
    #14 0xffff83147860 in draw_title_bar (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x7860)
    #15 0xffff83147f28 in draw_decoration (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x7f28)
    #16 0xffff83148190 in libdecor_plugin_gtk_frame_commit (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor/plugins-1/libdecor-gtk.so+0x8190)
    #17 0xffff86535708 in libdecor_frame_commit (/nix/store/q5a602jydkz3ina0pnaf32yb2zacch8v-libdecor-0.2.4/lib/libdecor-0.so.0+0x5708)
    #18 0xffff8f32fa90 in createShellObjects (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x2fa90)
    #19 0xffff8f316378 in glfwCreateWindow (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x16378)
    #20 0xaaaad46971a8 in createGLFWwindow /home/james/vsdf-present-fix/include/glfwutils.h:39
    #21 0xaaaad46971a8 in OnlineSDFRenderer::glfwSetup() /home/james/vsdf-present-fix/src/online_sdf_renderer.cpp:43
    #22 0xaaaad46ab790 in OnlineSDFRenderer::setup() /home/james/vsdf-present-fix/src/online_sdf_renderer.cpp:26
    #23 0xaaaad467e588 in run(int, char**) /home/james/vsdf-present-fix/src/main.cpp:370
    #24 0xaaaad46747ac in main /home/james/vsdf-present-fix/src/main.cpp:385
    #25 0xffff8b2b62b0 in __libc_start_call_main (/nix/store/89n0gcl1yjp37ycca45rn50h7lms5p6f-glibc-2.40-66/lib/libc.so.6+0x262b0) (BuildId: 70fbec6c13b6ae06e6b049bcbad20e1b16b0694e)
    #26 0xffff8b2b6394 in __libc_start_main@GLIBC_2.17 (/nix/store/89n0gcl1yjp37ycca45rn50h7lms5p6f-glibc-2.40-66/lib/libc.so.6+0x26394) (BuildId: 70fbec6c13b6ae06e6b049bcbad20e1b16b0694e)
    #27 0xaaaad4674fac in _start (/home/james/vsdf-present-fix/build/vsdf+0x34fac)

SUMMARY: AddressSanitizer: 538066 byte(s) leaked in 12086 allocation(s).

leakvsdf.log

Findings:

By running this on a WIP debug branch present-fix e7079f9 which honours the GLFW_PLATFORM

❯ GLFW_PLATFORM=x11 ./build/vsdf --toy plot.frag

now manages to work and get past the issue

It gets past the issue but now 1 more issue has been found:
#69

[jamylak]

Note even when setting

https://www.glfw.org/docs/latest/intro_guide.html#GLFW_WAYLAND_LIBDECOR_hint

to GLFW_WAYLAND_DISABLE_LIBDECOR it will still give these errors

Direct leak of 96 byte(s) in 1 object(s) allocated from:
    #0 0xffff84ab4d0c in calloc (/nix/store/j9nz3m8hqnyjjj5zxz5qvmd35g37rjyi-gcc-15.2.0-lib/lib/libasan.so.8+0xd4d0c)
    #1 0xffff7bbc6370 in proxy_create (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x6370) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #2 0xffff7bbc8818 in wl_proxy_marshal_array_flags (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x8818) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #3 0xffff7bbc8eec in wl_proxy_marshal_flags (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x8eec) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #4 0xffff84970244 in _glfwCreateWindowWayland (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x30244)
    #5 0xffff84956378 in glfwCreateWindow (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x16378)
    #6 0xaaaab95772bc in createGLFWwindow /home/james/vsdf-decor/include/glfwutils.h:40
    #7 0xaaaab95772bc in OnlineSDFRenderer::glfwSetup() /home/james/vsdf-decor/src/online_sdf_renderer.cpp:42
    #8 0xaaaab958c710 in OnlineSDFRenderer::setup() /home/james/vsdf-decor/src/online_sdf_renderer.cpp:25
    #9 0xaaaab955ea10 in run(int, char**) /home/james/vsdf-decor/src/main.cpp:449
    #10 0xaaaab9555094 in main /home/james/vsdf-decor/src/main.cpp:457
    #11 0xffff808d6570 in __libc_start_call_main (/nix/store/z6pz4znvbr103fln95y5xg1syb864x1i-glibc-2.42-47/lib/libc.so.6+0x26570) (BuildId: dddb2bd34273d7d23b99b7b0ecb1eda34d9e60c7)
    #12 0xffff808d6654 in __libc_start_main_impl (/nix/store/z6pz4znvbr103fln95y5xg1syb864x1i-glibc-2.42-47/lib/libc.so.6+0x26654) (BuildId: dddb2bd34273d7d23b99b7b0ecb1eda34d9e60c7)
    #13 0xaaaab95559ec in _start (/home/james/vsdf-decor/build/vsdf+0x359ec)

Direct leak of 96 byte(s) in 1 object(s) allocated from:
    #0 0xffff84ab4d0c in calloc (/nix/store/j9nz3m8hqnyjjj5zxz5qvmd35g37rjyi-gcc-15.2.0-lib/lib/libasan.so.8+0xd4d0c)
    #1 0xffff7bbc6370 in proxy_create (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x6370) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #2 0xffff7bbc8818 in wl_proxy_marshal_array_flags (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x8818) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #3 0xffff7bbc8eec in wl_proxy_marshal_flags (/nix/store/xcpdzp823kbj5qp5n3qgab3h8ny098fs-wayland-1.24.0/lib/libwayland-client.so.0+0x8eec) (BuildId: 366d1ece10bf99155de4f230a5b2de92c32010b4)
    #4 0xffff849702c0 in _glfwCreateWindowWayland (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x302c0)
    #5 0xffff84956378 in glfwCreateWindow (/nix/store/vw48ac82r2wg0ifkpn6v8330x2rx1grz-glfw-3.4/lib/libglfw.so.3+0x16378)
    #6 0xaaaab95772bc in createGLFWwindow /home/james/vsdf-decor/include/glfwutils.h:40
    #7 0xaaaab95772bc in OnlineSDFRenderer::glfwSetup() /home/james/vsdf-decor/src/online_sdf_renderer.cpp:42
    #8 0xaaaab958c710 in OnlineSDFRenderer::setup() /home/james/vsdf-decor/src/online_sdf_renderer.cpp:25
    #9 0xaaaab955ea10 in run(int, char**) /home/james/vsdf-decor/src/main.cpp:449
    #10 0xaaaab9555094 in main /home/james/vsdf-decor/src/main.cpp:457
    #11 0xffff808d6570 in __libc_start_call_main (/nix/store/z6pz4znvbr103fln95y5xg1syb864x1i-glibc-2.42-47/lib/libc.so.6+0x26570) (BuildId: dddb2bd34273d7d23b99b7b0ecb1eda34d9e60c7)
    #12 0xffff808d6654 in __libc_start_main_impl (/nix/store/z6pz4znvbr103fln95y5xg1syb864x1i-glibc-2.42-47/lib/libc.so.6+0x26654) (BuildId: dddb2bd34273d7d23b99b7b0ecb1eda34d9e60c7)
    #13 0xaaaab95559ec in _start (/home/james/vsdf-decor/build/vsdf+0x359ec)

SUMMARY: AddressSanitizer: 192 byte(s) leaked in 2 allocation(s).

but as mentioned before the X11 one works well, just need glfw 3.4 to set that hint

[jamylak]

After merging #74 now if you want to see the Wayland error just try

GLFW_PLATFORM=Wayland ./build/vsdf --toy example.frag --frames 1

and on a debug build it will print the leak. otherwise on release build without frames set, it will be unresponsive and hang