# XDP performance

**URL:** <https://community.solid-run.com/t/xdp-performance/567>\
**Category:** NXP LX2160\
**Created:** [January 31, 2023, 1:05pm UTC](https://community.solid-run.com/t/xdp-performance/567 "2023-01-31T13:05:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![agv100](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@agv100](https://community.solid-run.com/u/agv100)\
**Post date:** [January 31, 2023, 1:05pm UTC](https://community.solid-run.com/t/xdp-performance/567/1 "2023-01-31T13:05:04Z")

</div>

Dear SolidRun community,

Does anybody do any experients with XDP work on LX2160a board?  
As far as I can see, XDP Zero-copy support was added few months ago in Linux Vanilla kernel, and since August into 5.15 here [qoriq-components/linux - Linux Tree for QorIQ support](https://source.codeaurora.org/external/qoriq/qoriq-components/linux/log/?h=lf-5.15.y-rt) .

But, unfortunately, performance is mediocre.  
We’ve expected numbers, comparable with VPP-DPDK backed up (We still could not get it working stable), but in reality forwarding rate comparable with Linux Kernel:  
~ 500kpps for xdp\_router from tutorial  
~ 400kpps for VPP with AF\_XDP driver  
~370kpps for Linux Kernel  
~\>4.5Mpps for VPP/DPDK.

May be, we forgot something or do something wrong? If someone got satisfying performance, please, advice.  
Below are our setup/test details.

We use binary built by SolidRun lx2160a\_build, with simple changes to use 5.15 kernel from above.  
$ uname -a  
Linux localhost 5.15.71-rt51-07203-g00e98e11cb01 #1 SMP PREEMPT\_RT Fri Jan 27 22:25:43 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux  
$  
BPF/XDP related options are:

```auto
# grep -E "(BPF)|(XDP)" ./build/linux/.config
CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
# CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
# CONFIG_BPF_PRELOAD is not set
# CONFIG_BPF_LSM is not set
# end of BPF subsystem
CONFIG_CGROUP_BPF=y
CONFIG_XDP_SOCKETS=y
# CONFIG_XDP_SOCKETS_DIAG is not set
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
# CONFIG_BPFILTER is not set
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_BPF_STREAM_PARSER is not set
# CONFIG_SENSORS_XDPE122 is not set
CONFIG_BPF_EVENTS=y
# CONFIG_TEST_BPF is not set

```

eth1 and eth2 interfaces are created by ls-addni.

xdp-tutorial forwarding:  
[GitHub - xdp-project/xdp-tutorial: XDP tutorial](https://github.com/xdp-project/xdp-tutorial) built natively on board .  
xdp\_router microprogram attached.

```auto
../xdp-tutorial/packet-solutions# ./xdp_loader -d eth1 -N -F --progsec xdp_router
Success: Loaded BPF-object(xdp_prog_kern.o) and used section(xdp_router)
 - XDP prog attached on device:eth1(ifindex:5)
 - Pinning maps in /sys/fs/bpf/eth1/
../xdp-tutorial/packet-solutions# ./xdp_loader -d eth2 -N -F --progsec xdp_router
Success: Loaded BPF-object(xdp_prog_kern.o) and used section(xdp_router)
 - XDP prog attached on device:eth2(ifindex:6)
 - Pinning maps in /sys/fs/bpf/eth2/

```

As one can see below, program already attached in native mode. At least, ip link show indicated that.

```auto
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 06:1b:6f:31:a9:6c brd ff:ff:ff:ff:ff:ff
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d0:63:b4:03:1c:ce brd ff:ff:ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d0:63:b4:03:1c:cb brd ff:ff:ff:ff:ff:ff
    prog/xdp id 36 tag 551558afe8187df7 jited
6: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d0:63:b4:03:1c:cd brd ff:ff:ff:ff:ff:ff
    prog/xdp id 44 tag 551558afe8187df7 jited

```

Then, IP addresses are configured, ip forwarding sysctled to on (it has impact on bpf forwarding as well as on kernel forwarding), test performed by Cisco T-Rex.  
To be sure that forwarding goes through XDP, we can check that tcpdump does not see forwarded packets , we can also enable maps and see XDP\_REDIRECT counters increasing.  
Still, just 30% faster than kernel (while in examples over internet on PC boxes, it should be closer to DPDK).

VPP:  
same basic setup, then by vpp manual:

```auto
vpp# create interface af_xdp host-if eth1 num-rx-queues all
vpp# create interface af_xdp host-if eth2 num-rx-queues all

```

Then mac addresses are set, IP addreses are set and simple test performet by T-Rex with about 400kpps performance.  
Note, we were not able “tell” VPP to use zero-copy mode:

```auto
vpp# create interface af_xdp host-if eth2 num-rx-queues all zero-copy
af_xdp [error]: af_xdp_create_queue: xsk_socket__create() failed (is linux netdev eth2 up?): Operation not supported
create interface af_xdp: xsk_socket__create() failed (is linux netdev eth2 up?): Operation not supported
vpp#

```

During first “normal” interface creation, we also see some alarms in VPP logs:

```auto
libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Func#11 is safe for any args that match its prototype
btf_vmlinux is malformed
R1 type=ctx expected=fp
; int xdp_dispatcher(struct xdp_md *ctx)
0: (bf) r6 = r1
1: (b7) r0 = 2
; __u8 num_progs_enabled = conf.num_progs_enabled;
2: (18) r8 = 0xffff800008448000
4: (71) r7 = *(u8 *)(r8 +0)
 R0_w=invP2 R1=ctx(id=0,off=0,imm=0) R6_w=ctx(id=0,off=0,imm=0) R8_w=map_value(id=0,off=0,ks=4,vs=84,imm=0) R10=fp0
; if (num_progs_enabled < 1)
5: (15) if r7 == 0x0 goto pc+141
; ret = prog0(ctx);
6: (bf) r1 = r6
7: (85) call pc+140
btf_vmlinux is malformed
R1 type=ctx expected=fp
Caller passes invalid args into func#1
processed 84 insns (limit 1000000) max_states_per_insn 0 total_states 9 peak_states 9 mark_read 1
-- END PROG LOAD LOG --
libbpf: failed to load program 'xdp_dispatcher'
libbpf: failed to load object 'xdp-dispatcher.o'
libxdp: Failed to load dispatcher: Invalid argument
libxdp: Falling back to loading single prog without dispatcher

```

So, any ideas, what could be wrong?