Blog › ICP guides

IBM i developer on retainer: RPG IV ORDFULPRC CHAIN on INVMST acquires DB2 for i record lock on 847 inventory rows without UNLOCK after validation failure — IBM AS/400 developer, iSeries developer, IBM i developer on monthly retainer

October 10, 2026 · ~15 min read

An IBM i developer was maintaining the order fulfillment processing system for a wholesale building materials distributor operating 12 distribution warehouses across the US, running IBM i 7.5 on an IBM Power S924 server. The company’s inventory management and order processing systems were built entirely in ILE RPG (IBM’s Integrated Language Environment RPG IV), the fourth generation of the RPG (Report Program Generator) language that evolved from the original RPG for IBM System/360 in the 1960s through RPG II, RPG III, and into the ILE RPG used on every modern IBM i system. The order fulfillment batch job was ORDFULPRC — an ILE RPG program that ran each business day at 6:00 AM to process the previous day’s customer orders for warehouse fulfillment: for each order detail line, ORDFULPRC performed inventory availability validation, reserved stock, and generated a pick list record for the warehouse team. On a Thursday morning, the warehouse manager at the Memphis distribution center called the IT help desk: eight warehouse staff were waiting for their morning systems — order entry, inventory adjustment, and pick list programs — to respond. They had been waiting for 40 minutes.

ILE RPG accesses DB2 for i (formerly DB2/400 — the relational database engine embedded in the IBM i operating system itself, distinct from DB2 for z/OS on the mainframe) through file operations defined in the program’s F-specification (F-spec). The F-spec for ORDFULPRC’s inventory master file was: FINVMST    UF   A   E           K DISK            WAITRCD(*NOMAX). In ILE RPG F-spec syntax: UF is the file type “update full-procedural” (the program controls all reads explicitly — no automatic cycle reads — and the file is opened with both read and update capability); A is the file designation “add” (the program can add new records as well as read and update existing ones); E is the error-handling indicator (RPG IV handles file errors via the %ERROR built-in function rather than INFSR subroutine); K specifies keyed access (records accessed by key field rather than relative record number); DISK specifies a disk-based database file; and WAITRCD(*NOMAX) specifies that when waiting for a record lock held by another job, this program waits indefinitely with no timeout.

In ILE RPG, when a program performs a CHAIN operation on a file opened in UF (update) mode, the ILE RPG runtime issues a DB2 for i request to read the specified record and acquire an exclusive record-level lock on it — holding that record exclusively so that no other job can read it with update intent until this program either updates the record (UPDATE opcode), deletes it (DELETE opcode), explicitly releases the lock (UNLOCK opcode), reads the next record (READ/READE opcode, which releases the prior lock and acquires a lock on the new record), or the program ends (CLOSE file or program termination, which releases all held locks). ORDFULPRC’s processing loop CHAINed to INVMST using the item number from each order detail line: CHAIN  ORDLIN_ITEM_NBR  INVMST. This acquired an exclusive record lock on the INVMST (Inventory Master) physical file row for that item number. Then the program performed its validation logic:

First validation: IF  AVAIL_QTY < ORD_QTY (available quantity is less than the ordered quantity). If this condition was true, the order line could not be fulfilled from current stock. ORDFULPRC set an error flag and called ITER — the RPG IV opcode that advances to the next iteration of the enclosing DOW or FOR loop — to skip the fulfillment record generation and move to the next order line. ITER executed without an UNLOCK opcode before it. The exclusive DB2 for i record lock on the INVMST row for that item number remained held by job ORDFULFIL.

Second validation: IF  ORD_QTY < MIN_ORD_QTY (ordered quantity is below the minimum order quantity for that item). If this condition was true, ORDFULPRC also called ITER to skip and continue to the next line. Again, no UNLOCK before ITER. The exclusive DB2 for i record lock remained held on that INVMST row. Over 3,200 order detail lines processed that morning, 847 lines hit one of these two validation failures. Each time, a DB2 for i exclusive record lock was acquired by CHAIN and never released — not released by UNLOCK, not released by UPDATE (no update was done on failed lines), not released by a subsequent CHAIN to the same key (CHAIN to the same item number on a later order line would encounter the lock held by the same job and behave differently than a cross-job lock). After processing all 3,200 lines over approximately two hours, job ORDFULFIL held 847 exclusive record locks on INVMST simultaneously.

IBM AS/400, iSeries, and IBM i in 2026: the mid-range backbone of distribution, manufacturing, and banking

IBM announced the AS/400 (Application System/400) in June 1988 as the mid-range successor to the System/38, designed for the same business computing market as the System/36 but with a fundamentally different architecture: a single-level store addressing model, a database integrated into the operating system itself (the forerunner of DB2 for i), and a technology-independent machine interface (TIMI) that allowed programs to run unchanged across hardware generations. IBM renamed the platform iSeries in 2000, System i in 2006, and IBM i in 2008 when it consolidated the platform under the IBM Power Systems hardware brand. IBM i 7.5 (the current release as of 2026) runs on IBM Power S914, S924, E1080, and S1024 servers, all using IBM POWER10 processors. The operating system, the database, and the application layer are all integrated — DB2 for i is not a separately installed database server but a core component of IBM i OS, always running.

The IBM i installed base in 2026 is enormous among organizations that adopted the AS/400 in the 1990s and early 2000s: wholesale distribution companies (wholesale food, building materials, industrial supplies, pharmaceutical distribution), discrete and process manufacturers (automotive parts, chemical processing, consumer goods), regional and community banks (core banking on IBM i is common for banks under $5B in assets), insurance carriers (especially regional P/C and life carriers), and government agencies (state and local government systems built in the 1990s on RPG III or RPG IV). An IBM i developer on monthly retainer for a distribution company’s order management system maintains: the ILE RPG programs that process orders, manage inventory, and generate fulfillment records; the CL (Control Language) programs that submit jobs, manage library lists, and orchestrate batch streams; the DB2 for i physical files (tables) and logical files (views and indexes) that store and index the data; and the job scheduler entries that run the nightly and weekly batch programs automatically. The DB2 for i record lock model is central to this work because RPG IV’s file operations are designed around explicit lock acquisition and release — a model that rewards careful coding but punishes omissions like a missing UNLOCK before ITER.

The DB2 for i record lock model in ILE RPG works as follows: a physical file opened with UF (update full-procedural) usage gives the program the capability to both read records and update them. When the program performs a CHAIN, READ, READE, or READP to a UF file, DB2 for i acquires an exclusive record-level lock on the retrieved record. The exclusive lock means no other job can perform a CHAIN or READ for update on that same record until the lock is released. The lock is released by: UPDATE (writes the record back and releases the lock); DELETE (removes the record and releases the lock); UNLOCK (explicitly releases the lock without writing the record back — the record is unchanged, the lock is gone); the next sequential READ/READE/READP (in sequential reading mode, reading the next record releases the previous record’s lock and acquires a lock on the new record); CLOSE (closing the file releases all locks); or program termination (all files are implicitly closed). The WAITRCD keyword on the F-spec controls how long the program waits for a record that is currently locked by another job: WAITRCD(*IMMED) returns an error immediately if the record is locked; WAITRCD(30) waits up to 30 seconds; WAITRCD(*NOMAX) waits indefinitely with no timeout. ORDFULPRC’s F-spec had WAITRCD(*NOMAX) — if any other job tried to CHAIN to a record locked by ORDFULPRC, that job would wait forever.

The diagnostic command on IBM i for record lock analysis is WRKOBJLCK FILE(ITMLIB/INVMST) MBR(*FIRST) (Work with Object Locks). The result showed job ORDFULFIL (user ORDMGR, job type BCH, status ACTIVE/TIMW) holding 847 *EXCLRD (exclusive record) locks on the INVMST member. The warehouse staff jobs ORDENTER (order entry, user WHMGR01), INVADJ (inventory adjustment, user WHMGR02), and PICKLIST (pick list generation, user PICK01) appeared in lock wait status — WRKACTJOB showed their function as PGM-ORDMGR with status MTXW (mutex wait, indicating a lock wait). Because WAITRCD(*NOMAX) was specified on their programs’ F-specs for INVMST as well, none of these jobs would time out — they would wait for as long as ORDFULFIL held the locks, which could be another 30 minutes or more depending on how many more order lines remained to be processed.

ILE RPG record lock diagnosis: WRKOBJLCK, DSPJOBLOG, and the WAITRCD parameter

The IBM i IT administrator checked WRKACTJOB first, which showed ORDFULFIL in RUN status with elapsed time 2:17:33 — two hours and seventeen minutes, well past the expected 45-minute runtime for 3,200 order lines. WRKOBJLCK FILE(ITMLIB/INVMST) MBR(*FIRST) confirmed the 847 *EXCLRD locks held by ORDFULFIL. The administrator checked the DSPJOBLOG for ORDFULFIL, which showed no error messages — the job was progressing normally, just holding many locks. This is the signature of the ILE RPG CHAIN-without-UNLOCK pattern: the job itself is not failing or generating errors; it is simply executing its loop, and each validation failure quietly leaves behind a locked record with no log trace. Without monitoring specifically for WRKOBJLCK, the lock accumulation is invisible.

The IBM i developer reviewed ORDFULPRC’s source in QSYS.LIB (using WRKMBRPDM FILE(ORDLIB/ORDFULPSRC) to open the source member in SEU or RDi). The F-spec confirmed FINVMST UF A E K DISK WAITRCD(*NOMAX). The processing loop showed CHAIN ORDLIN_ITEM_NBR INVMST followed by two IF blocks that exited with ITER on validation failure — neither IF block contained an UNLOCK opcode before ITER. The pattern was confirmed in approximately 10 minutes of source review. The operational workaround was immediate: ENDJOB ORDFULFIL *CNTRLD (controlled end, which allows in-progress I/O to complete cleanly before ending the job). When ORDFULFIL ended, all 847 exclusive record locks were released instantly. The eight warehouse staff jobs that were in lock wait resumed within seconds as their blocked CHAIN requests succeeded.

The root cause was straightforward: the ORDFULPRC developer who wrote the validation logic added ITER to skip to the next order line on failure but did not add UNLOCK INVMST before the ITER. In RPG III (fixed-format, pre-ILE), the equivalent construct was a GOTO to the bottom of the calculation loop — developers in the RPG III era were more explicitly aware of lock release because the language required more explicit flow control. The migration to ILE RPG free-format procedures with structured DOW/FOR loops and ITER/LEAVE reduced the cognitive visibility of lock state. The fix was two lines of source: UNLOCK  INVMST added immediately before the ITER in each of the two validation failure branches. After recompiling ORDFULPRC with CRTRPGPGM PGM(ORDLIB/ORDFULPRC) SRCFILE(ORDLIB/ORDFULPSRC) and running a test batch with 3,200 representative order lines, WRKOBJLCK FILE(ITMLIB/INVMST) showed a maximum of one *EXCLRD lock at any time — the record currently being CHAINed and validated. All warehouse staff jobs ran concurrently with ORDFULPRC without any lock wait.

An additional hardening measure: the F-spec’s WAITRCD(*NOMAX) was changed to WAITRCD(60) as defense-in-depth. With WAITRCD(60), if any future code path leaves a lock held unintentionally, a waiting job will receive an error after 60 seconds (MCH3601 message code in the job log) rather than waiting indefinitely. The WAITRCD timeout generates a visible, log-searchable error that alerts operators to a lock conflict, whereas WAITRCD(*NOMAX) produces no log entry and no alert — jobs silently wait.

Typical IBM AS/400 developer retainer work and what it looks like in a work log

ILE RPG CHAIN without UNLOCK on validation failure accumulating hundreds of exclusive record locks is the canonical invisible IBM i lock bug for distribution and manufacturing companies whose batch order processing and inventory update programs were written with update-capable F-specs and ITER-based loop control. It appears in IBM i retainer work logs for wholesale distributor daily order fulfillment programs (CHAIN INVMST for each order line, ITER on stockout, no UNLOCK), manufacturer production order programs (CHAIN WORKINPROG for each production job, ITER on schedule conflict, no UNLOCK), bank account update batch programs (CHAIN ACCTMAST for each debit record, ITER on NSF check, no UNLOCK), and insurance claims posting programs (CHAIN CLAIMREC for each payment, ITER on coverage mismatch, no UNLOCK). The work log entry: “IBM i 7.5; job ORDFULFIL user ORDMGR BCH; program ORDFULPRC ILE RPG; F-spec FINVMST UF A E K DISK WAITRCD(*NOMAX); CHAIN ORDLIN_ITEM_NBR INVMST for each of 3,200 order detail lines; IF AVAIL_QTY < ORD_QTY → ITER (no UNLOCK INVMST); IF ORD_QTY < MIN_ORD_QTY → ITER (no UNLOCK INVMST); 847 validation failures across run → 847 *EXCLRD locks held 2h 17m; WRKOBJLCK FILE(ITMLIB/INVMST): 847 *EXCLRD locks; warehouse staff jobs ORDENTER, INVADJ, PICKLIST in MTXW lock wait; WAITRCD(*NOMAX) → indefinite wait, no log entry; ENDJOB ORDFULFIL *CNTRLD (operational workaround) releases all locks; fix: UNLOCK INVMST before each ITER at lines 118 and 143; CRTRPGPGM PGM(ORDLIB/ORDFULPRC); max concurrent locks: 847 → 1; also changed WAITRCD(*NOMAX) to WAITRCD(60) for defense-in-depth; hours: 1.5h diagnosis + 0.5h UNLOCK fix + 1h test.”

ILE RPG embedded SQL DECLARE CURSOR FOR UPDATE OF without COMMIT accumulating Db2 for i row locks is the IBM i equivalent of the mainframe Db2 batch cursor lock story. ILE RPG programs that use EXEC SQL DECLARE ORDHDR_CURSOR CURSOR FOR SELECT ORDER_NBR, CUST_NBR, ORDER_DATE FROM ORDHDR WHERE STATUS = 'PENDING' FOR UPDATE OF STATUS, PROC_DATE followed by FETCH in a loop, conditional UPDATE WHERE CURRENT OF on success paths, and ITER/LEAVE on validation failure paths without COMMIT accumulate Db2 for i row-level exclusive locks for the duration of the ILE RPG program. Work log: “IBM i 7.5; program ORDPOST ILE RPG embedded SQL; CRTSQLRPGI PGM(ORDLIB/ORDPOST); DECLARE ORDHDR_CURSOR CURSOR FOR SELECT ORDER_NBR, CUST_NBR FROM ORDHDR WHERE STATUS='PENDING' FOR UPDATE OF STATUS; FETCH NEXT FROM ORDHDR_CURSOR for 8,473 pending orders; EXEC SQL UPDATE ORDHDR SET STATUS='POSTED' WHERE CURRENT OF ORDHDR_CURSOR on success; FETCH without UPDATE on validation failure (customer credit hold: 312 orders); no EXEC SQL COMMIT between rows; 312 Db2 for i exclusive row locks held 1h 47m; online RPG order inquiry program ORDINQ CHAIN ORDHDR for 42 customers on credit hold receives lock wait; WAITRCD(30) → MCH3601 after 30s; 42 MCH3601 messages in ORDINQ job log; fix: EXEC SQL COMMIT every 500 FETCHes; EXEC SQL ROLLBACK on error exit path for credit hold orders (no partial credit hold update committed); MCH3601 rate: 0.”

ILE RPG data queue deadlock between order entry and inventory reservation programs is a common multi-program IBM i lock scenario for companies that use SNDDTAQ/RCVDTAQ for asynchronous communication between online order entry (interactive job) and background inventory reservation (batch job). If the batch reservation program holds a *EXCLRD lock on INVMST item ITM-004821 (via CHAIN) while waiting on a RCVDTAQ call, and the interactive order entry program holds a *EXCLRD lock on ORDHDR record for order 4791 (via CHAIN) while sending to the same SNDDTAQ, and the batch program needs to CHAIN ORDHDR to add an order reservation record — both jobs are waiting for each other. Work log: “IBM i 7.5; interactive job ORDERENT and batch job INVRSRV; ORDERENT CHAIN ORDHDR (lock held for edit session 14 minutes) + SNDDTAQ QORDER to trigger reservation; INVRSRV CHAIN INVMST ITM-004821 (lock held) + RCVDTAQ QORDER (waiting for message) + CHAIN ORDHDR ORDER-4791 (blocks, ORDERENT holds ORDHDR lock); ORDERENT waiting for ORDHDR read after reservation — WRKOBJLCK FILE INVMST: INVRSRV *EXCLRD; WRKOBJLCK FILE ORDHDR: ORDERENT *EXCLRD INVRSRV *EXCL request; deadlock; IBM i does not auto-detect application-level deadlocks (only DB2 for i SQL-layer deadlocks are auto-detected and rolled back); WRKACTJOB: both jobs in MTXW; ENDJOB INVRSRV *IMMED (operational break); fix: ORDERENT releases ORDHDR lock before SNDDTAQ (no pending edit session lock held during async wait); INVRSRV: CHAIN ORDHDR only after RCVDTAQ completes; deadlock rate: 0.”

How HourTab tracks IBM AS/400 developer retainer hours

The CHAIN-without-UNLOCK bug is invisible in unit testing by the same mechanism that makes all lock accumulation bugs invisible when only one job runs at a time. The ORDFULPRC developer who wrote the validation logic correctly used ITER to skip unfulfillable order lines — that was the right flow control decision for the business logic. The omission of UNLOCK before ITER was a lock management oversight, not a logic error. In development and QA environments where only one job runs ORDFULPRC at a time, no lock wait occurs, no error fires, and the program produces correct output. The bug emerges only in production, where warehouse staff jobs run concurrently with ORDFULPRC during the batch window that was originally designed to be exclusive but in practice overlaps with the 6:00 AM – 7:00 AM warehouse staff login period.

The work log entry that makes IBM i record lock work auditable must name every element of the causal chain: the IBM i OS version (IBM i 7.5), the program name and language (ORDFULPRC ILE RPG), the F-spec with USAGE attribute (FINVMST UF — update full-procedural), the opcode that acquired the lock (CHAIN), the validation failure condition that triggered ITER (AVAIL_QTY < ORD_QTY), the absence of UNLOCK before ITER, the lock accumulation count (847 *EXCLRD locks across 3,200 order lines), the diagnostic used (WRKOBJLCK FILE(ITMLIB/INVMST)), the blocked jobs (ORDENTER, INVADJ, PICKLIST), the WAITRCD parameter (WAITRCD(*NOMAX) — indefinite wait, no log entry), the fix (UNLOCK INVMST before ITER at lines 118 and 143), and the zero-lock-wait outcome. A log that says “fixed RPG lock issue, 3h” is not auditable to the distribution company’s warehouse operations manager who needs to explain to the regional VP why eight warehouse staff lost 40 minutes of productivity during morning receiving. HourTab gives IBM AS/400 developers a public retainer-hours URL they send to clients — wholesale distributors, manufacturers, banks, and insurance carriers that run critical daily IBM i batch programs and interactive order management systems maintained by an IBM i specialist on monthly retainer.

The broader context for IBM i retainer billing is that an ILE RPG CHAIN without UNLOCK on a validation failure path is the IBM i mid-range instantiation of the same principle that drives every exclusive lock retainer engagement across the mainframe and legacy enterprise stack. IBM Db2 developer retainers cover the same pattern at the z/OS mainframe level: INVUPD01 COBOL FOR UPDATE OF cursor accumulating 847,000 Db2 X locks without COMMIT over 3.5 hours — relational database X lock, same root cause. JCL developer retainers cover the same pattern at the z/OS dataset level: PREMBIL1 DISP=OLD holding z/OS ENQ EXCLUSIVE for 63 minutes, blocking PREMRPT1 — dataset-level lock, same class. CICS developer retainers cover the conversational VSAM exclusive record lock variant: ACMAINT READ UPDATE holding the CUSTFILE record lock for 23 minutes during CSR interaction — VSAM record lock. At each layer, the root cause is the same: a long-running exclusive reservation that was not released when the program took an early exit. IBM i’s RPG IV UNLOCK opcode, z/OS Db2’s COMMIT WORK, z/OS CICS’s pseudo-conversational redesign, and z/OS JCL’s DISP=SHR are each the correct, layer-specific release mechanism for the same category of problem.

FAQ: IBM AS/400 developer retainers