Add branches based on latest LTS kernels

[laijs]

While the v2 submission to the Linux kernel is not yet ready, two LTS branches based on the latest stable 6.6.x and 6.12.x will be created. Both branches will incorporate all development since the v1 submission, including several important ABI changes.

[bboymimi]

Hi @laijs,

I also spent some time last week forward-porting the PIE linking patches to the latest 6.12.34 LTS kernel. You can find my work here: https://github.com/bboymimi/linux/commits/6.12-pie-linking/

This is based on Ard's PIE linking patch set: https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=x86-pie-v3

Don't hesitate to let me know if you find any discrepancy. I'll continue helping with forward-porting the main PVM patches next.

[bboymimi]

Hi @laijs ,
During the forward-porting of the PVM patches to 6.12.34 LTS kernel, I encountered
the following issues:

• conflict: 48c825d ("KVM: x86: Implement gpc refresh for guest usage")
  • This looks a serious issue. The KVM_GUEST_USES_PFN sharing was removed from kernel because of the following commits:
  • a4bff3d ("KVM: pfncache: remove KVM_GUEST_USES_PFN usage")
  • eefb85b ("KVM: Drop unused @may_block param from gfn_to_pfn_cache_invalidate_start()")

I'm considering reverting the above two commits to restore the KVM_GUEST_USES_PFN
sharing. However, I'm not sure if this is the best solution. Given that you are the
PVM maintainer and have an extensive understanding of the PVM, I'd like to ask
for your opinion to have the best approach to resolve the issue.

[laijs]

@bboymimi It’s not ideal to base the rebase on Ard’s PIE patches that haven’t been merged yet. The proper approach is to rebase on 6.12.y, cherry-picking Ard’s patches that have already been accepted upstream, along with only the minimal additional patches needed for PVM.

[laijs]

48c825d ("KVM: x86: Implement gpc refresh for guest usage") addresses the concerns related to KVM_GUEST_USES_PFN, so the two commits that removed KVM_GUEST_USES_PFN can be reverted. However, this may not be the most suitable long-term solution for upstreaming PVM.

[bboymimi]

@bboymimi It’s not ideal to base the rebase on Ard’s PIE patches that haven’t been merged yet. The proper approach is to rebase on 6.12.y, cherry-picking Ard’s patches that have already been accepted upstream, along with only the minimal additional patches needed for PVM.

@laijs

Thanks for the prompt reply. In practice, I cherry-picked the patches one by one to 6.12.y (6.12.34) based on the Ard's PIE patches upstreaming status given by my script, which searches the title of the current upstream:

$ ./scan_non_upstream.sh 684a64bf32b6..ard/x86-pie-v3 linus/master                            
IN_SHA       UPSTREAM_SHA                             TAG          MESSAGE
------------ ---------------------------------------- ------------ ----------------------------------------
18f203506d7f e8fbc0d9cab6c1ee6403f42c0991b0c1d5dbc092 v6.13-rc1~202^2~5 x86/pvh: Call C code via the kernel virtual mapping
f893755af3f9 N/A                                      not-upstreamed Documentation: Bump minimum GCC version to 8.1
40a591f60458 N/A                                      not-upstreamed x86/tools: Use mmap() to simplify relocs host tool
0b1f15cc91fe N/A                                      not-upstreamed x86/boot: Permit GOTPCREL relocations for x86_64 builds
413a4ab08e92 N/A                                      not-upstreamed x86: Define the stack protector guard symbol explicitly
438770d53fb6 N/A                                      not-upstreamed x86/percpu: Get rid of absolute per-CPU variable placement
d3338cfba045 N/A                                      not-upstreamed scripts/kallsyms: Avoid 0x0 as the relative base
89cbbdfe0c8b N/A                                      not-upstreamed scripts/kallsyms: Remove support for absolute per-CPU variables
174a28d21ad9 N/A                                      not-upstreamed x86/tools: Remove special relocation handling for per-CPU variables
1f1a1a5b5eaa 223abe96ac0d227b22d48ab447dd9384b7a6c9fa v6.13-rc1~202^2~2 x86/xen: Avoid relocatable quantities in Xen ELF notes
efc5b475ef59 5c6808d1a9dd83853ff39684aeb812be39e108e8 v6.13-rc1~202^2~1 x86/pvh: Avoid absolute symbol references in .head.text
78d466eb54f3 N/A                                      not-upstreamed x86/pm-trace: Use RIP-relative accesses for .tracedata
f986d6af469a N/A                                      not-upstreamed x86/kvm: Use RIP-relative addressing
94e1a67d1b3b N/A                                      not-upstreamed x86/rethook: Use RIP-relative reference for return address
04a9fb1ec427 N/A                                      not-upstreamed x86/sync_core: Use RIP-relative addressing
901f0e40bcc4 N/A                                      not-upstreamed x86/entry_64: Use RIP-relative addressing
0d09176af06f N/A                                      not-upstreamed x86/hibernate: Prefer RIP-relative accesses
766d98d2868c 093562198e1a6360672954293753f4c6cb9a3316 v6.14-rc1~119^2~32 x86/boot/64: Determine VA/PA offset before entering C code
7b5dcc6cccf4 0d9b9a328cb605419ed046d341dc2a3d66ee0256 v6.14-rc1~119^2~31 x86/boot/64: Avoid intentional absolute symbol references in .head.text
ebcefc4e95d6 N/A                                      not-upstreamed x64/acpi: Use PIC-compatible references in wakeup_64.S
75b3dbc7e517 N/A                                      not-upstreamed x86/head: Use PIC-compatible symbol references in startup code
641ce548f60e N/A                                      not-upstreamed asm-generic: Treat PIC .data.rel.ro sections as .rodata
8f4cb76696ef N/A                                      not-upstreamed tools/objtool: Mark generated sections as writable
5e528762f3e8 N/A                                      not-upstreamed tools/objtool: Treat indirect ftrace calls as direct calls
47cf4a693042 N/A                                      not-upstreamed x86: Use PIE codegen for the core kernel
99d52d3ee2d6 N/A                                      not-upstreamed x86/boot: Implement support for ELF RELA/RELR relocations
831c7fd64d93 N/A                                      not-upstreamed x86/kernel: Switch to PIE linking for the core kernel

scan_non_upstream.sh

#!/bin/bash

# scan_non_upstream.sh - Find non-upstreamed commits by message match
# Usage: ./scan_non_upstream.sh <commit-range> <upstream-branch>
# Example: ./scan_non_upstream.sh 684a64bf32b6..ard/x86-pie-v3 pvm/6.12

set -e

usage() {
    echo "Usage: $0 <commit-range> <upstream-branch>"
    echo "Example: $0 684a64bf32b6..ard/x86-pie-v3 pvm/6.12"
    exit 1
}

if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
    usage
fi

if [ $# -ne 2 ]; then
    echo "Error: Invalid number of arguments."
    usage
fi

COMMIT_RANGE="$1"
UPSTREAM_BRANCH="$2"

# Print header
printf "%-12s %-40s %-12s %s\n" "IN_SHA" "UPSTREAM_SHA" "TAG" "MESSAGE"
printf "%-12s %-40s %-12s %s\n" "------------" "----------------------------------------" "------------" "----------------------------------------"

git log --reverse --pretty=format:'%h %s' "$COMMIT_RANGE" \
| while read in_sha msg; do
    # Find a commit in the upstream branch whose message matches exactly
    up_sha=$(git log "$UPSTREAM_BRANCH" --grep="^${msg}$" --pretty=format:'%H' -n1)
    if [ -n "$up_sha" ]; then
        tag=$(git describe --contains --abbrev=0 --match 'v[0-9]*' "$up_sha" 2>/dev/null || echo "no-tag")
    else
        tag="not-upstreamed"
    fi  
    printf "%-12s %-40s %-12s %s\n" "$in_sha" "${up_sha:-N/A}" "$tag" "$msg"
done 

I hope this is helpful. Please don't hesitate to let me know if I can help with anything.

[bysui]

Hi @bboymimi, Sorry for the late reply - we've been busy with our internal production environment.

Thanks for your branch to help with rebasing to the 6.12 LTS branch. We have pushed the 6.12 rebasing version to the branch pvm-612, which is based on 6.12.33. We have just tested some benchmarks to make sure they're working correctly. Also, it is compatible with the main branch (6.7.0-rc3). Welcome to give some feedback if there's anything wrong with the 6.12 branch.

[bboymimi]

@bysui , I have two issues to report after testing the PVM host by booting up into the Ubuntu desktop. The config is modified from the Ubuntu one. I found the BPF_JIT_ALWAYS_ON runtime is disabled on the new v6.12.33 kernel in your misc repo.

1. make -jnproc ARCH=x86_64 LLVM=1 bindeb-pkg 2>&1
   With LLVM=1, the build fails with the relocation error.

2. BPF_JIT_ALWAYS_ON runtime needs to be disabled or the following error will happen. I suspected the PTP classifier’s built‑in BPF program is invoking the JIT, and the x86 JIT can’t emit a reachable call (classic +-2 GiB rel32 overflow), or JIT memory can’t be made executable, and the init code uses BUG_ON(). Also, the options need to be put in kernel command line: "sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti"

3. By the way, have you tried bringing up the guest os by using an unmodified QEMU instead of the Kata container setup?

dmesg error:

[    0.375947] Target call (____ptrval____) is out of range
[    0.376098] Target call (____ptrval____) is out of range
[    0.376627] Target call (____ptrval____) is out of range
[    0.379742] bpf_jit: fatal error
[    0.380308] ------------[ cut here ]------------
[    0.380793] kernel BUG at net/core/ptp_classifier.c:227!
[    0.381092] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[    0.381662] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.12.33+ #17
[    0.382076] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[    0.382076] RIP: 0010:ptp_classifier_init+0x66/0x70
[    0.382076] Code: c2 29 00 48 89 45 f0 e8 98 58 c2 fe 85 c0 75 1a 48 8b 45 f8 65 48 2b 05 a8 08 54 00 75 0e c9 31 c0 31 f6 31 ff c3 cc cc cc cc <0f> 0b e8 d3 59 ee fe 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90
[    0.382076] RSP: 0000:ffffd0c500013d88 EFLAGS: 00010282
[    0.382076] RAX: fffffffffffffdf4 RBX: 0000000000000000 RCX: 0000000000000000
[    0.382076] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    0.382076] RBP: ffffd0c500013da0 R08: 0000000000000000 R09: 0000000000000000
[    0.382076] R10: 0000000000000000 R11: 0000000000000000 R12: ffffff87619cb6a0
[    0.382076] R13: ffffff8761ca0008 R14: ffffd0c500013df8 R15: ffffff8761e8345c
[    0.382076] FS:  0000000000000000(0000) GS:ffff8bba959a2000(0000) knlGS:0000000000000000
[    0.382076] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.382076] CR2: ffff8b41fffff000 CR3: 000000007961a001 CR4: 0000000000370ef0
[    0.382076] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.382076] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    0.382076] Call Trace:
[    0.382076]  <TASK>
[    0.382076]  sock_init+0xeb/0x110
[    0.382076]  ? __pfx_init_once+0x10/0x10
[    0.382076]  ? __pfx_sock_init+0x10/0x10
[    0.382076]  do_one_initcall+0x5e/0x330
[    0.382076]  kernel_init_freeable+0x358/0x530
[    0.382076]  ? __pfx_kernel_init+0x10/0x10
[    0.382076]  kernel_init+0x1c/0x200
[    0.382076]  ret_from_fork+0x48/0x70
[    0.382076]  ? __pfx_kernel_init+0x10/0x10
[    0.382076]  ret_from_fork_asm+0x1a/0x30
[    0.382076]  </TASK>
[    0.382076] Modules linked in:
[    0.382081] ---[ end trace 0000000000000000 ]---
[    0.382594] RIP: 0010:ptp_classifier_init+0x66/0x70
[    0.383079] Code: c2 29 00 48 89 45 f0 e8 98 58 c2 fe 85 c0 75 1a 48 8b 45 f8 65 48 2b 05 a8 08 54 00 75 0e c9 31 c0 31 f6 31 ff c3 cc cc cc cc <0f> 0b e8 d3 59 ee fe 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90

[bysui]

@bysui , I have two issues to report after testing the PVM host by booting up into the Ubuntu desktop. The config is modified from the Ubuntu one. I found the BPF_JIT_ALWAYS_ON runtime is disabled on the new v6.12.33 kernel in your misc repo.

1. make -jnproc ARCH=x86_64 LLVM=1 bindeb-pkg 2>&1
   With LLVM=1, the build fails with the relocation error.
2. BPF_JIT_ALWAYS_ON runtime needs to be disabled or the following error will happen. I suspected the PTP classifier’s built‑in BPF program is invoking the JIT, and the x86 JIT can’t emit a reachable call (classic +-2 GiB rel32 overflow), or JIT memory can’t be made executable, and the init code uses BUG_ON(). Also, the options need to be put in kernel command line: "sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti"
3. By the way, have you tried bringing up the guest os by using an unmodified QEMU instead of the Kata container setup?

dmesg error:

[ 0.375947] Target call (ptrval) is out of range
[ 0.376098] Target call (ptrval) is out of range
[ 0.376627] Target call (ptrval) is out of range
[ 0.379742] bpf_jit: fatal error
[ 0.380308] ------------[ cut here ]------------
[ 0.380793] kernel BUG at net/core/ptp_classifier.c:227!
[ 0.381092] Oops: invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 0.381662] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.12.33+ #17
[ 0.382076] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 0.382076] RIP: 0010:ptp_classifier_init+0x66/0x70
[ 0.382076] Code: c2 29 00 48 89 45 f0 e8 98 58 c2 fe 85 c0 75 1a 48 8b 45 f8 65 48 2b 05 a8 08 54 00 75 0e c9 31 c0 31 f6 31 ff c3 cc cc cc cc <0f> 0b e8 d3 59 ee fe 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90
[ 0.382076] RSP: 0000:ffffd0c500013d88 EFLAGS: 00010282
[ 0.382076] RAX: fffffffffffffdf4 RBX: 0000000000000000 RCX: 0000000000000000
[ 0.382076] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 0.382076] RBP: ffffd0c500013da0 R08: 0000000000000000 R09: 0000000000000000
[ 0.382076] R10: 0000000000000000 R11: 0000000000000000 R12: ffffff87619cb6a0
[ 0.382076] R13: ffffff8761ca0008 R14: ffffd0c500013df8 R15: ffffff8761e8345c
[ 0.382076] FS: 0000000000000000(0000) GS:ffff8bba959a2000(0000) knlGS:0000000000000000
[ 0.382076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.382076] CR2: ffff8b41fffff000 CR3: 000000007961a001 CR4: 0000000000370ef0
[ 0.382076] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.382076] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.382076] Call Trace:
[ 0.382076]
[ 0.382076] sock_init+0xeb/0x110
[ 0.382076] ? __pfx_init_once+0x10/0x10
[ 0.382076] ? __pfx_sock_init+0x10/0x10
[ 0.382076] do_one_initcall+0x5e/0x330
[ 0.382076] kernel_init_freeable+0x358/0x530
[ 0.382076] ? __pfx_kernel_init+0x10/0x10
[ 0.382076] kernel_init+0x1c/0x200
[ 0.382076] ret_from_fork+0x48/0x70
[ 0.382076] ? __pfx_kernel_init+0x10/0x10
[ 0.382076] ret_from_fork_asm+0x1a/0x30
[ 0.382076]
[ 0.382076] Modules linked in:
[ 0.382081] ---[ end trace 0000000000000000 ]---
[ 0.382594] RIP: 0010:ptp_classifier_init+0x66/0x70
[ 0.383079] Code: c2 29 00 48 89 45 f0 e8 98 58 c2 fe 85 c0 75 1a 48 8b 45 f8 65 48 2b 05 a8 08 54 00 75 0e c9 31 c0 31 f6 31 ff c3 cc cc cc cc <0f> 0b e8 d3 59 ee fe 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90

Hi, @bboymimi. Thanks for your quick feedback! It seems that I need to do more tests :).

For issue 1, I have just tested guest kernel building on LLVM (my local LLVM version is 20.1.7), and I encountered two problems. One is that there's a section type mismatch for the .relr.auth.dyn section. I'm not an expert in the compiler area; I just googled that the .relr.auth.dyn section is used for pointer authentication in LLVM, which is only supported on AArch64. So I'm not sure where and why it is generated—I'm still studying this. I didn't encounter it on the GNU ld side because my binutils version is 2.36, so it seems that it doesn't support the '-z pack-relative-reloc' option (I found it is supported since version 2.38). Therefore, only the .rela section is generated, not the .relr section. I would like to try updating my binutils to see if there is a similar problem with GNU ld. Another problem is that LLD complains about relocation in a readonly section—that is, .Lcommon_startup_64 is defined in the __INITRODATA section. This is checked by the '-z text' option; it seems that this option has some differences between GNU LD and LLD. However, I think we can change it by defining Lcommon_startup_64 in the __INITDATA section instead to solve this problem. I'm not sure if your problems on LLVM are the same as mine.

For issue 2, I used the config file from Kata Containers (vmlinux-6.12.36-160), and turned on the CONFIG_PVM_GUEST option, so it seems that CONFIG_BPF_JIT_ALWAYS_ON is not enabled in the Kata Containers guest config file. I have just turned on the CONFIG_BPF_JIT_ALWAYS_ON option, but I didn't encounter your problem. So could you provide your guest config file?

For issue 3, after removing KVM_CPUID_VENDOR_FEATURES in KVM (see commit 24879a4 in the pvm branch), QEMU doesn't need any changes now, except for snapshot support which needs saving/restoring PVM virtual MSRs. I used the QEMU built by myself without any code changes in my test environment now and didn't see any problems. Could you elaborate more on your problems?

[bboymimi]

@bysui,

For issue one, it's exactly the same issue. The Lcommon_startup_64 is in
INITRODATA section. After using gcc, the error has disappeared.

For issue two, I enabled PIE in the host kernel as well. So, it happens when
CONFIG_BPF_JIT_ALWAYS_ON is enabled. You can try to enable it and see if the
error still happens. Even if I disabled it, the aforementioned kernel command:
"sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti"
still needs to be placed to avoid triggering the error. This is my config.

For issue three, sorry that I didn't describe the problem clearly. My problem is
that I still couldn't successfully bring up the guest VMs. Even I tried the
kernel/image inside the cloud image you guys released in the
pvm-get-started-with-kata.md:
https://github.com/virt-pvm/misc/releases/download/test/pvm-kata-vm-img.tar.gz

What I found on the internet is all about the firecracker and kata container.
I'm trying to make it easy for a general public to try, especially in Ubuntu
community. Currently, I've built the debian package and the host kernel can be
brought up successfully into the Ubuntu Desktop. However, I tried to
reconstruct the Qemu command from the Kata container setup in the mentioned
cloud image, but it failed. I'm not sure if the problem is with the Qemu or the
kernel.

sudo /usr/bin/qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -bios "$PWD/mnt/opt/kata/share/kata-qemu/qemu/qboot.rom" \
  -cpu host,pmu=off -m 2048 -smp 1 -nographic \
  -kernel "$PWD/mnt/opt/kata/share/kata-containers/vmlinux.pvm" \
  -append "console=ttyS0 root=/dev/sda1 rw" \
  -drive if=none,id=drv0,file="$PWD/mnt/opt/kata/share/kata-containers/kata-containers.img",format=raw,aio=io_uring \
  -device virtio-scsi-pci,id=scsi0 \
  -device scsi-hd,drive=drv0,bus=scsi0.0 \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0,romfile=

It could be possible that the 6.12.33 PVM host kernel is not compatible with the
6.7 PVM guest. If you could provide the detailed instructions on how you build
the host+guest and how to bring it up manually by Qemu, it would be very helpful
for me to help with the testing.

Afterward, if we make the PVM ported into the Ubuntu kernel, I believe there
will have a sensational impact on the PVM project.

[bysui]

@bysui,

For issue one, it's exactly the same issue. The Lcommon_startup_64 is in INITRODATA section. After using gcc, the error has disappeared.

Yes, I have just moved Lcommon_startup_64 to the INITDATA section for now. And as for the unexpected .relr.auth.dyn section, I changed the linker script to ignore it, which follows the rule in the arm64 linker script. After those changes, I can build the guest kernel on LLVM successfully and boot it successfully too.

--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -234,7 +234,7 @@ SECTIONS

        .init.relr : {
                __relr_start = .;
-               *(.relr.*)
+               *(.relr.dyn)
                __relr_end = .;

For issue two, I enabled PIE in the host kernel as well. So, it happens when CONFIG_BPF_JIT_ALWAYS_ON is enabled. You can try to enable it and see if the error still happens. Even if I disabled it, the aforementioned kernel command: "sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti" still needs to be placed to avoid triggering the error. This is my config.

Thanks for your config. Actually, I don't suggest enabling PIE on the host kernel either, as PIE building for x86 kernel is not ideal for now, which increases the text size in some cases and needs more help from the compiler. However, I will try your config file for the host kernel later to see the problem.

For issue three, sorry that I didn't describe the problem clearly. My problem is that I still couldn't successfully bring up the guest VMs. Even I tried the kernel/image inside the cloud image you guys released in the pvm-get-started-with-kata.md: https://github.com/virt-pvm/misc/releases/download/test/pvm-kata-vm-img.tar.gz

What I found on the internet is all about the firecracker and kata container. I'm trying to make it easy for a general public to try, especially in Ubuntu community. Currently, I've built the debian package and the host kernel can be brought up successfully into the Ubuntu Desktop. However, I tried to reconstruct the Qemu command from the Kata container setup in the mentioned cloud image, but it failed. I'm not sure if the problem is with the Qemu or the kernel.

sudo /usr/bin/qemu-system-x86_64
-machine q35,accel=kvm
-bios "$PWD/mnt/opt/kata/share/kata-qemu/qemu/qboot.rom"
-cpu host,pmu=off -m 2048 -smp 1 -nographic
-kernel "$PWD/mnt/opt/kata/share/kata-containers/vmlinux.pvm"
-append "console=ttyS0 root=/dev/sda1 rw"
-drive if=none,id=drv0,file="$PWD/mnt/opt/kata/share/kata-containers/kata-containers.img",format=raw,aio=io_uring
-device virtio-scsi-pci,id=scsi0
-device scsi-hd,drive=drv0,bus=scsi0.0
-netdev user,id=net0,hostfwd=tcp::2222-:22
-device virtio-net-pci,netdev=net0,romfile=
It could be possible that the 6.12.33 PVM host kernel is not compatible with the 6.7 PVM guest. If you could provide the detailed instructions on how you build the host+guest and how to bring it up manually by Qemu, it would be very helpful for me to help with the testing.

Before pushing the 6.12.33 branch, I had tested all combinations of 6.12.33 PVM host/guest with 6.7 PVM host/guest. Sorry, we had updated some ABIs between guest/host. Are you using the new 6.7 branch for guest/host, as I see that you use the /mnt/opt/kata/share/kata-containers/vmlinux.pvm option in your QEMU command line? I had updated the VM image last week, which doesn't have this file anymore. There are vmlinux-pvm-6.12.33 and vmlinux-pvm-6.7.0-rc6—these two guest kernels are in this directory.

Afterward, if we make the PVM ported into the Ubuntu kernel, I believe there will have a sensational impact on the PVM project.

Good, we can try it together. :)

[bboymimi]

@bysui

Sorry for the late response. I was off last week.

@bysui,
For issue one, it's exactly the same issue. The Lcommon_startup_64 is in INITRODATA section. After using gcc, the error has disappeared.

Yes, I have just moved Lcommon_startup_64 to the INITDATA section for now. And as for the unexpected .relr.auth.dyn section, I changed the linker script to ignore it, which follows the rule in the arm64 linker script. After those changes, I can build the guest kernel on LLVM successfully and boot it successfully too.

--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -234,7 +234,7 @@ SECTIONS

        .init.relr : {
                __relr_start = .;
-               *(.relr.*)
+               *(.relr.dyn)
                __relr_end = .;

It looks good to me.

For issue two, I enabled PIE in the host kernel as well. So, it happens when CONFIG_BPF_JIT_ALWAYS_ON is enabled. You can try to enable it and see if the error still happens. Even if I disabled it, the aforementioned kernel command: "sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti" still needs to be placed to avoid triggering the error. This is my config.

Thanks for your config. Actually, I don't suggest enabling PIE on the host kernel either, as PIE building for x86 kernel is not ideal for now, which increases the text size in some cases and needs more help from the compiler. However, I will try your config file for the host kernel later to see the problem.

Thank you for the reminder. I've built the PIE host kernel for testing purposes. It seems to be working well with the PIE guest.

For issue three, sorry that I didn't describe the problem clearly. My problem is that I still couldn't successfully bring up the guest VMs. Even I tried the kernel/image inside the cloud image you guys released in the pvm-get-started-with-kata.md: https://github.com/virt-pvm/misc/releases/download/test/pvm-kata-vm-img.tar.gz
What I found on the internet is all about the firecracker and kata container. I'm trying to make it easy for a general public to try, especially in Ubuntu community. Currently, I've built the debian package and the host kernel can be brought up successfully into the Ubuntu Desktop. However, I tried to reconstruct the Qemu command from the Kata container setup in the mentioned cloud image, but it failed. I'm not sure if the problem is with the Qemu or the kernel.
sudo /usr/bin/qemu-system-x86_64
-machine q35,accel=kvm
-bios "$PWD/mnt/opt/kata/share/kata-qemu/qemu/qboot.rom"
-cpu host,pmu=off -m 2048 -smp 1 -nographic
-kernel "$PWD/mnt/opt/kata/share/kata-containers/vmlinux.pvm"
-append "console=ttyS0 root=/dev/sda1 rw"
-drive if=none,id=drv0,file="$PWD/mnt/opt/kata/share/kata-containers/kata-containers.img",format=raw,aio=io_uring
-device virtio-scsi-pci,id=scsi0
-device scsi-hd,drive=drv0,bus=scsi0.0
-netdev user,id=net0,hostfwd=tcp::2222-:22
-device virtio-net-pci,netdev=net0,romfile=
It could be possible that the 6.12.33 PVM host kernel is not compatible with the 6.7 PVM guest. If you could provide the detailed instructions on how you build the host+guest and how to bring it up manually by Qemu, it would be very helpful for me to help with the testing.

Before pushing the 6.12.33 branch, I had tested all combinations of 6.12.33 PVM host/guest with 6.7 PVM host/guest. Sorry, we had updated some ABIs between guest/host. Are you using the new 6.7 branch for guest/host, as I see that you use the /mnt/opt/kata/share/kata-containers/vmlinux.pvm option in your QEMU command line? I had updated the VM image last week, which doesn't have this file anymore. There are vmlinux-pvm-6.12.33 and vmlinux-pvm-6.7.0-rc6—these two guest kernels are in this directory.

Exactly, I've tried the new updated cloud image. It works well with the pre-built PVM guest. As mentioned before, I also built the PVM guest. Both are working well.

Afterward, if we make the PVM ported into the Ubuntu kernel, I believe there will have a sensational impact on the PVM project.

Good, we can try it together. :)

The other performance issue I observed is when using the bpftrace first time in the guest PVM, it's a bit slow, hanging for several seconds before printing out the message. Then, it works fine afterward. The command is as follows:
sudo BPFTRACE_MAX_STRLEN=200 bpftrace -q -e 'BEGIN { printf("banner: %s\n", str(kaddr("linux_banner"))); exit() }'

The other question: have you tried the Windows VM? I was thinking about another scenario that runs the Windows VM with a designated application as the light VM to replace Wine.

[bysui]

@bysui

Sorry for the late response. I was off last week.

@bysui,
For issue one, it's exactly the same issue. The Lcommon_startup_64 is in INITRODATA section. After using gcc, the error has disappeared.

Yes, I have just moved Lcommon_startup_64 to the INITDATA section for now. And as for the unexpected .relr.auth.dyn section, I changed the linker script to ignore it, which follows the rule in the arm64 linker script. After those changes, I can build the guest kernel on LLVM successfully and boot it successfully too.

--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -234,7 +234,7 @@ SECTIONS

        .init.relr : {
                __relr_start = .;
-               *(.relr.*)
+               *(.relr.dyn)
                __relr_end = .;

It looks good to me.

For issue two, I enabled PIE in the host kernel as well. So, it happens when CONFIG_BPF_JIT_ALWAYS_ON is enabled. You can try to enable it and see if the error still happens. Even if I disabled it, the aforementioned kernel command: "sysctl.net.core.bpf_jit_enable=0 initcall_blacklist=ptp_classifier_init nopti" still needs to be placed to avoid triggering the error. This is my config.

Thanks for your config. Actually, I don't suggest enabling PIE on the host kernel either, as PIE building for x86 kernel is not ideal for now, which increases the text size in some cases and needs more help from the compiler. However, I will try your config file for the host kernel later to see the problem.

Thank you for the reminder. I've built the PIE host kernel for testing purposes. It seems to be working well with the PIE guest.

For issue three, sorry that I didn't describe the problem clearly. My problem is that I still couldn't successfully bring up the guest VMs. Even I tried the kernel/image inside the cloud image you guys released in the pvm-get-started-with-kata.md: https://github.com/virt-pvm/misc/releases/download/test/pvm-kata-vm-img.tar.gz
What I found on the internet is all about the firecracker and kata container. I'm trying to make it easy for a general public to try, especially in Ubuntu community. Currently, I've built the debian package and the host kernel can be brought up successfully into the Ubuntu Desktop. However, I tried to reconstruct the Qemu command from the Kata container setup in the mentioned cloud image, but it failed. I'm not sure if the problem is with the Qemu or the kernel.
sudo /usr/bin/qemu-system-x86_64
-machine q35,accel=kvm
-bios "$PWD/mnt/opt/kata/share/kata-qemu/qemu/qboot.rom"
-cpu host,pmu=off -m 2048 -smp 1 -nographic
-kernel "$PWD/mnt/opt/kata/share/kata-containers/vmlinux.pvm"
-append "console=ttyS0 root=/dev/sda1 rw"
-drive if=none,id=drv0,file="$PWD/mnt/opt/kata/share/kata-containers/kata-containers.img",format=raw,aio=io_uring
-device virtio-scsi-pci,id=scsi0
-device scsi-hd,drive=drv0,bus=scsi0.0
-netdev user,id=net0,hostfwd=tcp::2222-:22
-device virtio-net-pci,netdev=net0,romfile=
It could be possible that the 6.12.33 PVM host kernel is not compatible with the 6.7 PVM guest. If you could provide the detailed instructions on how you build the host+guest and how to bring it up manually by Qemu, it would be very helpful for me to help with the testing.

Before pushing the 6.12.33 branch, I had tested all combinations of 6.12.33 PVM host/guest with 6.7 PVM host/guest. Sorry, we had updated some ABIs between guest/host. Are you using the new 6.7 branch for guest/host, as I see that you use the /mnt/opt/kata/share/kata-containers/vmlinux.pvm option in your QEMU command line? I had updated the VM image last week, which doesn't have this file anymore. There are vmlinux-pvm-6.12.33 and vmlinux-pvm-6.7.0-rc6—these two guest kernels are in this directory.

Exactly, I've tried the new updated cloud image. It works well with the pre-built PVM guest. As mentioned before, I also built the PVM guest. Both are working well.

Afterward, if we make the PVM ported into the Ubuntu kernel, I believe there will have a sensational impact on the PVM project.

Good, we can try it together. :)

The other performance issue I observed is when using the bpftrace first time in the guest PVM, it's a bit slow, hanging for several seconds before printing out the message. Then, it works fine afterward. The command is as follows: sudo BPFTRACE_MAX_STRLEN=200 bpftrace -q -e 'BEGIN { printf("banner: %s\n", str(kaddr("linux_banner"))); exit() }'

Sorry, I haven't looked into it until now. I will take a look next week. By the way, did it happen in both the 6.7 and 6.12 guests?

The other question: have you tried the Windows VM? I was thinking about another scenario that runs the Windows VM with a designated application as the light VM to replace Wine.
No, the x86 instruction emulator in KVM is incomplete; some important instructions are not supported (e.g., IRET in long mode). Even if we implement them in the emulator, performance would be poor, so we need to modify the guest to replace those instructions with hypercalls for better performance. Therefore, Windows VMs are not supported for now.

Furthermore, we are actually more inclined toward Wine rather than VM, as emulating full hardware is not our goal. I recall that @laijs had a plan to expand the usage of the PVM switcher in Wine.