{"id":2120,"date":"2024-11-12T11:29:43","date_gmt":"2024-11-12T11:29:43","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=2120"},"modified":"2025-12-01T04:48:47","modified_gmt":"2025-12-01T04:48:47","slug":"deploying-lustre-file-system-with-rdma-node-maps-and-acls","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/deploying-lustre-file-system-with-rdma-node-maps-and-acls\/","title":{"rendered":"Deploying Lustre File System with RDMA, Node Maps, and ACLs"},"content":{"rendered":"\n

Lustre is the de facto parallel file system for high-performance computing (HPC) clusters, providing extreme scalability, high throughput, and low-latency access across thousands of nodes. This guide walks through a complete deployment of Lustre using RDMA over InfiniBand<\/strong> for performance, along with Node Maps<\/strong> for client access control and ACLs<\/strong> for fine-grained permissions.<\/p>\n\n\n\n

\n\n\n\n

1. Understanding the Lustre Architecture<\/h2>\n\n\n\n

Lustre separates metadata and data services into distinct roles:<\/p>\n\n\n\n

    \n
  • MGS (Management Server)<\/strong> \u2013 Manages Lustre configuration and coordinates cluster services.<\/li>\n\n\n\n
  • MDT (Metadata Target)<\/strong> \u2013 Stores file system metadata (names, permissions, directories).<\/li>\n\n\n\n
  • OST (Object Storage Target)<\/strong> \u2013 Stores file data blocks.<\/li>\n\n\n\n
  • Clients<\/strong> \u2013 Mount and access the Lustre file system for I\/O.<\/li>\n<\/ul>\n\n\n\n

    The typical architecture looks like this:<\/p>\n\n\n\n

    +-------------+        +-------------+\n|   Client 1  |        |   Client 2  |\n| \/mnt\/lustre |        | \/mnt\/lustre |\n+------+------+        +------+------+\n       |                        |\n       +--------o2ib RDMA-------+\n                |\n        +-------+-------+\n        |     OSS\/OST    |\n        |   (Data I\/O)   |\n        +-------+-------+\n                |\n        +-------+-------+\n        |     MGS\/MDT    |\n        |  (Metadata)    |\n        +---------------+\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    2. Prerequisites and Environment<\/h2>\n\n\n\n
    Component<\/th>Requirements<\/th><\/tr><\/thead>
    OS<\/strong><\/td>RHEL \/ Rocky \/ AlmaLinux 8.x or higher<\/td><\/tr>
    Kernel<\/strong><\/td>Built with Lustre and OFED RDMA modules<\/td><\/tr>
    Network<\/strong><\/td>InfiniBand fabric (Mellanox or compatible)<\/td><\/tr>
    Lustre Version<\/strong><\/td>2.14 or later<\/td><\/tr>
    Devices<\/strong><\/td>Separate block devices for MDT, OST(s), and client mount<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    3. Install Lustre Packages<\/h2>\n\n\n\n
    On MGS, MDT, and OSS Nodes:<\/h3>\n\n\n\n
    dnf install -y lustre kmod-lustre lustre-osd-ldiskfs\n<\/code><\/pre>\n\n\n\n
    On Client Nodes:<\/h3>\n\n\n\n
    dnf install -y lustre-client kmod-lustre-client\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    4. Configure InfiniBand and RDMA (o2ib)<\/h2>\n\n\n\n
    InfiniBand provides the lowest latency for Lustre communication via RDMA. Configure the o2ib<\/code> network type for Lustre.<\/p>\n\n\n\n
    1. Install and verify InfiniBand stack<\/h3>\n\n\n\n
    dnf install -y rdma-core infiniband-diags perftest libibverbs-utils\nsystemctl enable --now rdma\nibstat\n<\/code><\/pre>\n\n\n\n
    2. Configure IB network<\/h3>\n\n\n\n
    nmcli con add type infiniband ifname ib0 con-name ib0 ip4 10.0.0.1\/24\nnmcli con up ib0\n<\/code><\/pre>\n\n\n\n
    3. Verify RDMA link<\/h3>\n\n\n\n
    ibv_devinfo\nibv_rc_pingpong -d mlx5_0\n<\/code><\/pre>\n\n\n\n
    4. Configure LNET for o2ib<\/h3>\n\n\n\n
    Create \/etc\/modprobe.d\/lustre.conf<\/code> with:<\/p>\n\n\n\n
    options lnet networks=\"o2ib(ib0)\"\n<\/code><\/pre>\n\n\n\n
    modprobe lnet\nlnetctl lnet configure\nlnetctl net add --net o2ib --if ib0\nlnetctl net show\n<\/code><\/pre>\n\n\n\n
    Expected output:<\/p>\n\n\n\n
    net:\n  - net type: o2ib\n    interfaces:\n      0: ib0\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    5. Format and Mount Lustre Targets<\/h2>\n\n\n\n
    Metadata Server (MGS + MDT)<\/h3>\n\n\n\n
    mkfs.lustre --fsname=lustrefs --mgs --mdt --index=0 \/dev\/sdb\nmount -t lustre \/dev\/sdb \/mnt\/mdt\n<\/code><\/pre>\n\n\n\n
    Object Storage Server (OSS)<\/h3>\n\n\n\n
    mkfs.lustre --fsname=lustrefs --ost --index=0 --mgsnode=<MGS>@o2ib \/dev\/sdc\nmount -t lustre \/dev\/sdc \/mnt\/ost\n<\/code><\/pre>\n\n\n\n
    Client Node<\/h3>\n\n\n\n
    mount -t lustre <MGS>@o2ib:\/lustrefs \/mnt\/lustre\nsudo mkdir -p \/mnt\/lustre\n\nsudo mount -t lustre \\\n  172.16.0.10@o2ib:\/lustrefs \\\n  \/mnt\/lustre\n\nexample without ibnetwork\n[root@vbox ~]# mount -t lustre 172.16.0.10@tcp:\/lustre \/mnt\/lustre-client\n[root@vbox ~]# \n[root@vbox ~]# # Verify the mount worked\n[root@vbox ~]# df -h \/mnt\/lustre-client\nFilesystem                Size  Used Avail Use% Mounted on\n172.16.0.10@tcp:\/lustre   12G  2.5M   11G   1% \/mnt\/lustre-client\n[root@vbox ~]# lfs df -h\nUUID                       bytes        Used   Available Use% Mounted on\nlustre-MDT0000_UUID         4.5G        1.9M        4.1G   1% \/mnt\/lustre-client[MDT:0]\nlustre-OST0000_UUID         7.5G        1.2M        7.0G   1% \/mnt\/lustre-client[OST:0]\nlustre-OST0001_UUID         3.9G        1.2M        3.7G   1% \/mnt\/lustre-client[OST:1]\nfilesystem_summary:        11.4G        2.4M       10.7G   1% \/mnt\/lustre-client\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    6. Configuring Node Maps (Access Control)<\/h2>\n\n\n\n
    Node maps allow administrators to restrict Lustre client access based on network or host identity.<\/p>\n\n\n\n
    1. View current node maps<\/h3>\n\n\n\n
    lctl nodemap_list\n<\/code><\/pre>\n\n\n\n
    2. Create a new node map for trusted clients<\/h3>\n\n\n\n
    lctl nodemap_add trusted_clients\n<\/code><\/pre>\n\n\n\n
    3. Add allowed network range or host<\/h3>\n\n\n\n
    lctl nodemap_add_range trusted_clients 10.0.0.0\/24\n<\/code><\/pre>\n\n\n\n
    4. Enable enforcement<\/h3>\n\n\n\n
    lctl set_param nodemap.trusted_clients.admin=1\nlctl set_param nodemap.trusted_clients.trust_client_ids=1\n<\/code><\/pre>\n\n\n\n
    5. Restrict default map<\/h3>\n\n\n\n
    lctl set_param nodemap.default.reject_unauthenticated=1\n<\/code><\/pre>\n\n\n\n
    This ensures only IPs in 10.0.0.0\/24<\/code> can mount and access the Lustre filesystem.<\/p>\n\n\n\n
    \n\n\n\n
    7. Configuring Access Control Lists (ACLs)<\/h2>\n\n\n\n
    Lustre supports standard POSIX ACLs for fine-grained directory and file permissions.<\/p>\n\n\n\n
    1. Enable ACL support on mount<\/h3>\n\n\n\n
    mount -t lustre -o acl <MGS>@o2ib:\/lustrefs \/mnt\/lustre\n<\/code><\/pre>\n\n\n\n
    2. Verify ACL support<\/h3>\n\n\n\n
    mount | grep lustre\n<\/code><\/pre>\n\n\n\n
    Should show:<\/p>\n\n\n\n
    \/dev\/sda on \/mnt\/lustre type lustre (rw,acl)\n<\/code><\/pre>\n\n\n\n
    3. Set ACLs on directories<\/h3>\n\n\n\n
    setfacl -m u:researcher:rwx \/mnt\/lustre\/projects\nsetfacl -m g:analysts:rx \/mnt\/lustre\/reports\n<\/code><\/pre>\n\n\n\n
    4. View ACLs<\/h3>\n\n\n\n
    getfacl \/mnt\/lustre\/projects\n<\/code><\/pre>\n\n\n\n
    Sample output:<\/p>\n\n\n\n
    # file: projects\n# owner: root\n# group: root\nuser::rwx\nuser:researcher:rwx\ngroup::r-x\ngroup:analysts:r-x\nmask::rwx\nother::---\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    8. Verifying Cluster Health<\/h2>\n\n\n\n
    On all nodes:<\/h3>\n\n\n\n
    lctl ping <MGS>@o2ib\nlctl dl\nlctl get_param -n net.*.state\n<\/code><\/pre>\n\n\n\n
    Check RDMA performance:<\/h3>\n\n\n\n
    lctl get_param -n o2iblnd.*.stats\n<\/code><\/pre>\n\n\n\n
    Check file system mount from client:<\/h3>\n\n\n\n
    df -h \/mnt\/lustre\n<\/code><\/pre>\n\n\n\n
    Optional: Check node map enforcement<\/h3>\n\n\n\n
    Try mounting from an unauthorized IP \u2014 it should fail:<\/p>\n\n\n\n
    mount -t lustre <MGS>@o2ib:\/lustrefs \/mnt\/test\nmount.lustre: mount <MGS>@o2ib:\/lustrefs at \/mnt\/test failed: Permission denied\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    9. Common Issues and Troubleshooting<\/h2>\n\n\n\n
    Issue<\/th>Possible Cause<\/th>Resolution<\/th><\/tr><\/thead>
    Mount failed: no route to host<\/code><\/td>IB subnet mismatch or LNET not configured<\/td>Verify lnetctl net show<\/code> and ping -I ib0<\/code> between nodes.<\/td><\/tr>
    Permission denied<\/code><\/td>Node map restriction active<\/td>Check lctl nodemap_list<\/code> and ensure client IP range is allowed.<\/td><\/tr>
    Slow performance<\/code><\/td>RDMA disabled or fallback to TCP<\/td>Verify lctl list_nids<\/code> shows @o2ib<\/code> transport.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    10. Final Validation Checklist<\/h2>\n\n\n\n