Hi there,

I successfully set up iSCSI server based on tgtd. It uses iscsi driver,
no authorization, no problems found.
Since I use InfiniBand I decided to switch to iSER (iSCSI over RDMA). Unfortunately, after changing driver to iser I get the following:

on the client side:
# iscsiadm -m node -l
iscsiadm: Could not login to [iface: default, target: iqn.2024-09.org.open-iscsi.hera:iscsivolumea, portal: 192.168.0.129,3260]. iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI
login failure)
iscsiadm: Could not log into all portals

on the server side:
tgtd: iser_device_init(3259) dev 0x55bbffc24b60
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed
tgtd: iser_init_rdma_buf_pool(647) pool buf:0x7fc1cc2b7000
list:0x55bbffc24d50 mr:0x55bbffc40d60 lkey:0x17ffff
tgtd: iser_device_init(3278) max 524288 CQEs
tgtd: iser_device_init(3295) CQ vector: 0
tgtd: iser_device_init(3303) dev->cq:0x55bbffc40db0
tgtd: iser_cm_connect_request(1493) alloc conn:0x55bbffc410b0 cm_id:0x55bbffc24110
tgtd: iser_conn_login_phase_set(874) conn:0x55bbffc410b0 from:INIT to:START tgtd: iser_cm_connect_request(1552) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, created qp:0x55bbffc41688
tgtd: iser_alloc_login_resources(895) conn:0x55bbffc410b0 login tasks
num:2, buf_sz:8544 (rx_sz:8524 tx_sz:8524)
tgtd: iser_alloc_login_resources(944) post_recv login rx task:0x55bbffc73330 tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:0 + 1 = 1
tgtd: iser_alloc_login_resources(951) saved login tx-task:0x55bbffc73888
tgtd: iser_conn_get(1371) refcnt:2
tgtd: iser_cm_connect_request(1586) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, accepted
tgtd: iser_cm_conn_established(1613) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, established
tgtd: handle_wc(2956) recv complete, wr_id:0x55bbffc73390 len:560
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_rx_handler_non_ff(2784) login rx, conn:0x55bbffc410b0
tgtd: iser_login_exec(681) conn:0x55bbffc410b0 Login request
(operational negotiation): 15
tgtd: iser_login_rx(2536) conn:0x55bbffc410b0, login failed, class:2
detail:3
tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:1 + 1 = 2
tgtd: schedule_resp_tx(156) task:0x55bbffc73888 tag:0x0000 cmdsn:0x0
tgtd: iser_prep_resp_send_req(395) task:0x55bbffc73888
wr_id:0x55bbffc73998 tag:0x0000 dtbuf:0x55bbffc7120c dtsz:0 ahs_sz:0
stat:0x0 statsn:0x0 expcmdsn:0x0
tgtd: iser_post_send(495) conn:0x55bbffc410b0 posted:1 1st
wr:0x55bbffc73998 wr_id:0x55bbffc73998 sge_sz:76
tgtd: iser_conn_getn(1378) refcnt:2 + 1 = 3
tgtd: handle_wc(2956) send complete, wr_id:0x55bbffc73998 len:21947
tgtd: iser_tx_complete_handler(2893) conn:0x55bbffc410b0
task:0x55bbffc73888 tag:0x0000 opcode:0x23
tgtd: iser_conn_put(1386) refcnt:2
tgtd: iser_complete_task(831) task:0x55bbffc73888, non-cmd
tgtd: iser_cm_disconnected(1633) conn:0x55bbffc410b0
cm_id:0x55bbffc24110 event:10, RDMA_CM_EVENT_DISCONNECTED
tgtd: iser_conn_close(1277) rdma_disconnect conn:0x55bbffc410b0
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_conn_close(1292) conn:0x55bbffc410b0 cm_id:0x0x55bbffc24110
state: CLOSE, refcnt:2
tgtd: handle_wc_error(2997) conn:0x55bbffc410b0 task:0x55bbffc73330
tag:0x0000 wr_id:0x0x55bbffc73390 op:recv err:Work Request Flushed Error vendor_err:0xf5
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_cm_timewait_exit(1648) conn:0x55bbffc410b0 refcnt:1 cm_id:0x55bbffc24110
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_cm_timewait_exit(1666) conn:0x55bbffc410b0 refcnt:1 cm_id:0x55bbffc24110 (after cleanup)
tgtd: iser_conn_put(1386) refcnt:0
tgtd: iser_conn_free(1315) conn:0x55bbffc410b0 refcnt:0
qp:0x55bbffc41688 cm_id:0x55bbffc24110
tgtd: iser_ib_clear_iosubmit_list(1179) start
tgtd: iser_ib_clear_rdma_rd_list(1193) start
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_ib_clear_sent_list(1221) start
tgtd: iser_ib_clear_post_recv_list(1234) start
tgtd: iser_free_login_resources(967) conn:0x55bbffc410b0, login phase:START tgtd: iser_conn_free(1356) conn:0x55bbffc410b0 freed

Which is very strange. First, I do not have authorization, and second, I
also use NFS over RDMA and its working fine.

Any ideas?

PS. Server is Trixi and client is Sid

--- PyGate Linux v1.5.10
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On Sun, 1 Feb 2026, Greg wrote:

Hi there,

I successfully set up iSCSI server based on tgtd. It uses iscsi driver, no authorization, no problems found.
Since I use InfiniBand I decided to switch to iSER (iSCSI over RDMA). Unfortunately, after changing driver to iser I get the following:

on the client side:
# iscsiadm -m node -l
iscsiadm: Could not login to [iface: default, target: iqn.2024-09.org.open-iscsi.hera:iscsivolumea, portal: 192.168.0.129,3260]. iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals

on the server side:
tgtd: iser_device_init(3259) dev 0x55bbffc24b60
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed
tgtd: iser_init_rdma_buf_pool(647) pool buf:0x7fc1cc2b7000 list:0x55bbffc24d50 mr:0x55bbffc40d60 lkey:0x17ffff
tgtd: iser_device_init(3278) max 524288 CQEs
tgtd: iser_device_init(3295) CQ vector: 0
tgtd: iser_device_init(3303) dev->cq:0x55bbffc40db0
tgtd: iser_cm_connect_request(1493) alloc conn:0x55bbffc410b0 cm_id:0x55bbffc24110
tgtd: iser_conn_login_phase_set(874) conn:0x55bbffc410b0 from:INIT to:START tgtd: iser_cm_connect_request(1552) conn:0x55bbffc410b0 cm_id:0x55bbffc24110,
created qp:0x55bbffc41688
tgtd: iser_alloc_login_resources(895) conn:0x55bbffc410b0 login tasks num:2, buf_sz:8544 (rx_sz:8524 tx_sz:8524)
tgtd: iser_alloc_login_resources(944) post_recv login rx task:0x55bbffc73330 tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:0 + 1 = 1
tgtd: iser_alloc_login_resources(951) saved login tx-task:0x55bbffc73888 tgtd: iser_conn_get(1371) refcnt:2
tgtd: iser_cm_connect_request(1586) conn:0x55bbffc410b0 cm_id:0x55bbffc24110,
192.168.0.130 -> 192.168.0.129, accepted
tgtd: iser_cm_conn_established(1613) conn:0x55bbffc410b0 cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, established
tgtd: handle_wc(2956) recv complete, wr_id:0x55bbffc73390 len:560
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_rx_handler_non_ff(2784) login rx, conn:0x55bbffc410b0
tgtd: iser_login_exec(681) conn:0x55bbffc410b0 Login request (operational negotiation): 15
tgtd: iser_login_rx(2536) conn:0x55bbffc410b0, login failed, class:2 detail:3 tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:1 + 1 = 2
tgtd: schedule_resp_tx(156) task:0x55bbffc73888 tag:0x0000 cmdsn:0x0
tgtd: iser_prep_resp_send_req(395) task:0x55bbffc73888 wr_id:0x55bbffc73998 tag:0x0000 dtbuf:0x55bbffc7120c dtsz:0 ahs_sz:0 stat:0x0 statsn:0x0 expcmdsn:0x0
tgtd: iser_post_send(495) conn:0x55bbffc410b0 posted:1 1st wr:0x55bbffc73998 wr_id:0x55bbffc73998 sge_sz:76
tgtd: iser_conn_getn(1378) refcnt:2 + 1 = 3
tgtd: handle_wc(2956) send complete, wr_id:0x55bbffc73998 len:21947
tgtd: iser_tx_complete_handler(2893) conn:0x55bbffc410b0 task:0x55bbffc73888 tag:0x0000 opcode:0x23
tgtd: iser_conn_put(1386) refcnt:2
tgtd: iser_complete_task(831) task:0x55bbffc73888, non-cmd
tgtd: iser_cm_disconnected(1633) conn:0x55bbffc410b0 cm_id:0x55bbffc24110 event:10, RDMA_CM_EVENT_DISCONNECTED
tgtd: iser_conn_close(1277) rdma_disconnect conn:0x55bbffc410b0
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_conn_close(1292) conn:0x55bbffc410b0 cm_id:0x0x55bbffc24110 state:
CLOSE, refcnt:2
tgtd: handle_wc_error(2997) conn:0x55bbffc410b0 task:0x55bbffc73330 tag:0x0000 wr_id:0x0x55bbffc73390 op:recv err:Work Request Flushed Error vendor_err:0xf5
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_cm_timewait_exit(1648) conn:0x55bbffc410b0 refcnt:1 cm_id:0x55bbffc24110
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_cm_timewait_exit(1666) conn:0x55bbffc410b0 refcnt:1 cm_id:0x55bbffc24110 (after cleanup)
tgtd: iser_conn_put(1386) refcnt:0
tgtd: iser_conn_free(1315) conn:0x55bbffc410b0 refcnt:0 qp:0x55bbffc41688 cm_id:0x55bbffc24110
tgtd: iser_ib_clear_iosubmit_list(1179) start
tgtd: iser_ib_clear_rdma_rd_list(1193) start
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_ib_clear_sent_list(1221) start
tgtd: iser_ib_clear_post_recv_list(1234) start
tgtd: iser_free_login_resources(967) conn:0x55bbffc410b0, login phase:START tgtd: iser_conn_free(1356) conn:0x55bbffc410b0 freed

Which is very strange. First, I do not have authorization, and second, I also
use NFS over RDMA and its working fine.

Any ideas?

PS. Server is Trixi and client is Sid

No idea but I'd start here:
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed

And I see this in the README.iser on github for tgt:

2.2. Memory registration

One of the most severe stumbling blocks in moving any application to
take advantage of RDMA features is memory registration. Before using
RDMA, both the sending and receiving buffers must be registered with
the operating system. This operation ensures that the underlying
hardware pages will not be modified during the transfer, and provides
the physical addresses of the buffers to the network card. However,
the process itself is time consuming, and CPU intensive. Previous investigations have shown that for InfiniBand, the throughput drops
by up to 40% when memory registration and deregistration are included
in the critical path.

This iSER implementation uses pre-registered buffers for RDMA
operations. In general such a scheme is difficult to justify due
to the large per-connection resource requirements. However, in
this application it may be appropriate. Since the target always
initiates RDMA operations and never advertises RDMA buffers, it can
securely use one pool of buffers for multiple clients and can manage
its memory resources explicitly. Also, the architecture of the code
is such that the iSCSI layer dictates incoming and outgoing buffer
locations to the storage device layer, so supplying a registered
buffer is relatively easy.

Perhaps you need to have some other config somewhere setting this up?

tgt is quite old in debian. I think the maintainer is MIA. Upstream is responsive though, there's a couple of patches of mine upstream (which
aren't relevant to your issue) and no doubt there are many others that
might be relevant - debian is a dozen or so versions behind, at least in trixie.

Tim.

--- PyGate Linux v1.5.10
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On 2/1/26 18:02, Tim Woodall wrote:

No idea but I'd start here:
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed

According to this
https://lists.wpkg.org/pipermail/stgt/2013-July/019036.html
the error is harmless

--- PyGate Linux v1.5.10
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On Sun, 1 Feb 2026, Greg wrote:

On 2/1/26 18:02, Tim Woodall wrote:

No idea but I'd start here:
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed

According to this
https://lists.wpkg.org/pipermail/stgt/2013-July/019036.html
the error is harmless

That link doesn't work for me but assuming that it's the same thread as
the one (right year and month) I found then it says:

I deleted the targets and performed discovery and it works fine now.
Even without pool_sz_mb=100.

--- PyGate Linux v1.5.10
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On 2/1/26 19:45, Tim Woodall wrote:

On Sun, 1 Feb 2026, Greg wrote:

On 2/1/26 18:02, Tim Woodall wrote:

No idea but I'd start here:
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed

According to this
https://lists.wpkg.org/pipermail/stgt/2013-July/019036.html
the error is harmless

That link doesn't work for me but assuming that it's the same thread as
the one (right year and month) I found then it says:

I deleted the targets and performed discovery and it works fine now.
Even without pool_sz_mb=100.

The message also mentions rebuilding from sources. So, it looks that it
was a tgtd bug. Probably fixed years ago since the thread is from 2013.

--- PyGate Linux v1.5.10
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)