Error in compiling OAT trampoline lib

[teozoia]

I am interested in reproducing your work, I've first compile OAT with:

cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_ASSERTIONS=On -DLLVM_INCLUDE_TESTS=OFF -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-none-elf -DLLVM_TARGET_ARCH=AArch64 -DLLVM_TARGETS_TO_BUILD=AArch64  ../oat-llvm40/

When I try to compile the trampoline library:

$ cd oat-trampoline-lib
$ make lib-arch64
clang  -I/usr/aarch64-linux-gnu/include -fPIC -g -c -Wall nova.c
clang  -I/usr/aarch64-linux-gnu/include -fPIC -g -c -Wall cfv_bellman.c
cfv_bellman.c:206:43: warning: value size does not match register size specified
      by the constraint and modifier [-Wasm-operand-widths]
        asm volatile("msr pmcr_el0, %0" : : "r" (17));
                                                 ^
cfv_bellman.c:206:30: note: use constraint modifier "w"
        asm volatile("msr pmcr_el0, %0" : : "r" (17));
                                    ^~
                                    %w0
cfv_bellman.c:208:49: warning: value size does not match register size specified
      by the constraint and modifier [-Wasm-operand-widths]
        asm volatile("msr PMCNTENSET_EL0, %0" : : "r" (0x8000000f));
                                                       ^
cfv_bellman.c:208:36: note: use constraint modifier "w"
        asm volatile("msr PMCNTENSET_EL0, %0" : : "r" (0x8000000f));
                                          ^~
                                          %w0
cfv_bellman.c:210:47: warning: value size does not match register size specified
      by the constraint and modifier [-Wasm-operand-widths]
        asm volatile("msr PMOVSCLR_EL0, %0" : : "r" (0x8000000f));
                                                     ^
cfv_bellman.c:210:34: note: use constraint modifier "w"
        asm volatile("msr PMOVSCLR_EL0, %0" : : "r" (0x8000000f));
                                        ^~
                                        %w0
cfv_bellman.c:217:45: warning: value size does not match register size specified
      by the constraint and modifier [-Wasm-operand-widths]
        asm volatile("mrs %0, PMCCNTR_EL0" : "=r" (cc));
                                                   ^
cfv_bellman.c:217:20: note: use constraint modifier "w"
        asm volatile("mrs %0, PMCCNTR_EL0" : "=r" (cc));
                          ^~
                          %w0
4 warnings generated.

There are some errors in the linker:

clang  -I/usr/aarch64-linux-gnu/include -fPIC -g -c -Wall trampoline.S
clang  -I/usr/aarch64-linux-gnu/include -L/usr/aarch64-linux-gnu/lib -shared -Wl,-soname,libnova.so.1 -o libnova.so.1.0.1 nova.o trampoline.o cfv_bellman.o -lc

ld: Unknown command line argument '-plugin'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-debug'?
ld: Unknown command line argument '-plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper'?
ld: Unknown command line argument '-plugin-opt=-fresolution=/tmp/ccUosbZ3.res'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-fresolution=/tmp/ccUosbZ3.res'?
ld: Unknown command line argument '-plugin-opt=-pass-through=-lgcc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-pass-through=-lgcc'?
ld: Unknown command line argument '-plugin-opt=-pass-through=-lgcc_s'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-pass-through=-lgcc_s'?
ld: Unknown command line argument '-plugin-opt=-pass-through=-lc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-pass-through=-lc'?
ld: Unknown command line argument '-plugin-opt=-pass-through=-lgcc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-pass-through=-lgcc'?
ld: Unknown command line argument '-plugin-opt=-pass-through=-lgcc_s'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=-pass-through=-lgcc_s'?
ld: Unknown command line argument '--sysroot=/'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-import=/'?
ld: Unknown command line argument '--build-id'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-help-list'?
ld: Unknown command line argument '--eh-frame-hdr'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-override'?
ld: Unknown command line argument '-m'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '--hash-style=gnu'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-rng-seed=gnu'?
ld: Unknown command line argument '--as-needed'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-only-needed'?
ld: Unknown command line argument '-shared'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-stats'?
ld: Unknown command line argument '-z'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '-L/usr/aarch64-linux-gnu/lib'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-summary-index'?
ld: Unknown command line argument '-L/usr/aarch64-linux-gnu/include/../lib'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-asm-macro-max-nesting-depth'?
ld: Unknown command line argument '-L/usr/lib/gcc/x86_64-linux-gnu/5'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-L/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-L/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-L/lib/x86_64-linux-gnu'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-L/lib/../lib'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-help-list'?
ld: Unknown command line argument '-L/usr/lib/x86_64-linux-gnu'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-L/usr/lib/../lib'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-help-list'?
ld: Unknown command line argument '-L/usr/aarch64-linux-gnu/include'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-ignore-empty-index-file'?
ld: Unknown command line argument '-L.'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '-L/usr/lib/gcc/x86_64-linux-gnu/5/../../..'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-disable-force-link-odr'?
ld: Unknown command line argument '-soname'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-stats'?
ld: Unknown command line argument '-lc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '-lgcc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '--as-needed'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-only-needed'?
ld: Unknown command line argument '-lgcc_s'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-stats'?
ld: Unknown command line argument '--no-as-needed'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-only-needed'?
ld: Unknown command line argument '-lc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '-lgcc'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-S'?
ld: Unknown command line argument '--as-needed'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-only-needed'?
ld: Unknown command line argument '-lgcc_s'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-stats'?
ld: Unknown command line argument '--no-as-needed'.  Try: '/usr/bin/ld -help'
ld: Did you mean '-only-needed'?
collect2: error: ld returned 1 exit status
clang-4.0: error: linker (via gcc) command failed with exit code 1 (use -v to see invocation)
Makefile:30: recipe for target 'lib-aarch64' failed
make: *** [lib-aarch64] Error 1

There is something missing?

[sunzc]

Where did you compile the trampoline lib?
I compiled on a hikey board(with Arm64 Arch native support), otherwise you should set up a cross-compile environment for ARM64 binary if you are on x86-64 environment(Intel/AMD server).

[teozoia]

As you can see from the first compile step I'm cross-compiling (with llvm cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_ASSERTIONS=On -DLLVM_INCLUDE_TESTS=OFF -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-none-elf -DLLVM_TARGET_ARCH=AArch64 -DLLVM_TARGETS_TO_BUILD=AArch64 ../oat-llvm40/) the trampoline library for aarch64 since I don't have an Hikey board.

llvm runs on x86_64 Intel but is producing code for aarch64.

[sunzc]

Can you try to compile it from a native Aarch64(Arm64) board or QEMU-Arm? I'm assuming you need such an environment to repeat the experiment, not necessarily Hikey board.

I did not write CMake file for the directory, and I don't have first-hand experience for cross-compiling the files.

[zz-fz-john]

I think it is a fault about linker,Your linker maybe is gcc that targets x86. I use -v parameters.

clang -shared -Wl,-soname,libnova.so.1 -o libnova.so.1.0.1 nova.o trampoline.o cfv_bellman.o -lc -v
clang version 4.0.0 (tags/RELEASE_400/final)
Target: aarch64-none--elf
Thread model: posix
InstalledDir: /home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin
 "/usr/lib/ccache/gcc" -shared -v -o libnova.so.1.0.1 -Wl,-soname,libnova.so.1 nova.o trampoline.o cfv_bellman.o -lc
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.5.0-12ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.5.0 20171010 (Ubuntu 5.5.0-12ubuntu1) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/5/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-shared' '-v' '-o' 'libnova.so.1.0.1' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/5/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/5/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper -plugin-opt=-fresolution=/tmp/ccKm1VPJ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -shared -z relro -o libnova.so.1.0.1 /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/5/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/5/../../.. -soname libnova.so.1 nova.o trampoline.o cfv_bellman.o -lc -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/5/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
/usr/bin/ld: nova.o：普通ELF重定位（M: 183）
nova.o: 无法添加符号: 文件格式错误
collect2: error: ld returned 1 exit status
clang-4.0: error: linker (via gcc) command failed with exit code 1 (use -v to see invocation)

So I think you could change the linker.

[zz-fz-john]

I solved this error.In makefile ,Modify as follows：

TEE_PATH ?= /home/zhaoruizhe517/optee2.5hikeydebian
TOOLCHAIN ?= $(TEE_PATH)/toolchains/aarch64/bin/
TA ?= $(TEE_PATH)/hello_world/ta
TEEC_EXPORT ?= $(TEE_PATH)/optee_client/out/export
CFLAGS += -Wall -I$(TA)/include -I$(TEEC_EXPORT)/include -I./include
LDADD += -lteec -L$(TEEC_EXPORT)/lib
CC = /home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin/clang
TARGET = -target aarch64-unknown-linux
CCC = $(TOOLCHAIN)/aarch64-linux-gnu-gcc
SOFT_CFLAGS=-Wall -pedantic  -O3 -D__SOFTBOUNDCETS_SPATIAL
ARFLAGS=-rcs 
lib-aarch64:
	/home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin/clang  -fPIC -g -c -Wall nova.c
	/home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin/clang  -fPIC -g -c -Wall cfv_bellman.c
	/home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin/clang  -fPIC -g -c -Wall trampoline.S
	/home/zhaoruizhe517/OAT/OAT-Project/oat-llvm40/build/bin/clang -shared --target=aarch64-linux-gnu-gcc -Wl,-soname,libnova.so.1 -o libnova.so.1.0.1 nova.o trampoline.o cfv_bellman.o -lc
	sudo cp libnova.so.1.0.1 /usr/lib/
	sudo ln -fs libnova.so.1.0.1 /usr/lib/libnova.so.1
	sudo ln -fs /usr/lib/libnova.so.1 /usr/lib/libnova.so