COOL:2E developer on retainer: PGORDCHK Change function type generates FORDMST UF update-mode F-spec locking ORDMST record during ORDENT screen I/O while warehouse ORDPICK waits

COOL:2E 6.0 (Synon/2E) • ARCAD ALM • IBM i 7.4 • ILE RPG generated • DB2 for i record lock • Function Type • retainer billing

Consolidated Wholesale Distributors had been running their warehouse and order management systems on IBM i for twenty-two years. The core applications were built in COOL:2E Version 5, migrated to COOL:2E 6.0, and never converted out of the 4GL model. The original COOL:2E developers were long gone. When the warehouse complained that the order picking system was dying with cryptic “machine check” errors every few hours, the IT manager brought in a COOL:2E retainer consultant who had been working with Synon/2E and its successors since the 1990s.

The symptom: job ORDPICK, running under user profile WHPICK, would terminate abnormally with MCH3601 (object not available) during peak order processing. It happened only when order entry operators were actively processing orders. It never happened during off-hours batch runs. Classic interactive-vs-batch lock contention — but the actual source was buried in how COOL:2E had generated a retrieval-only function with the wrong access mode.

COOL:2E function types and generated F-specs

COOL:2E generates ILE RPG source code from a model. One of the most consequential modeling decisions is the function type. COOL:2E function types include:

The critical difference: Change and Delete generate UF (update-full) in the F-spec. Any CHAIN on a UF file acquires an exclusive DB2 for i record lock (*EXCLRD). That lock is held until either UPDATE or UNLOCK is executed. If the program returns or branches to a non-update exit without UNLOCK, the lock is held in the calling activation group until that group ends.

The PGORDCHK mismodeled function

PGORDCHK (Order Check) was a validation function called from PGORDENT (Order Entry), the interactive order entry display program. Its design intent was simple: given an order number, retrieve the ORDMST (Order Master) record and check whether the order had already been picked — if so, return a DUPLICATE-ORDER return code to prevent the operator from double-picking it.

The original developer had modeled PGORDCHK as a Change function type, probably because Change was the default when creating functions that accessed ORDMST in other contexts. The COOL:2E model generated this RPG F-spec for PGORDCHK:

FORDMST    UF   A   E           K DISK    WAITRDC(60)
*                ^^^ Update-Full = CHAIN acquires *EXCLRD lock

The generated action diagram logic was:

C     ORDNO         CHAIN     ORDMST               /* *EXCLRD lock acquired */
C                   IF        %FOUND(ORDMST)
C                   IF        ORD_STATUS = 'PICKED'
C                   EVAL      RTNCOD = 'DUPORD'    /* return DUPLICATE-ORDER */
C                   RETURN                         /* exits — *EXCLRD still held */
C                   ENDIF
C                   ENDIF
C                   EVAL      RTNCOD = 'OK'
C                   RETURN

PGORDCHK never contained an UPDATE ORDMST call — it was read-only. But the Change function type caused COOL:2E to generate a UF F-spec regardless. The CHAIN at line 1 acquired a *EXCLRD record lock on the ORDMST record. The RETURN at the DUPLICATE-ORDER exit returned control to PGORDENT — but in an *CALLER activation group, the lock from PGORDCHK stayed open in PGORDENT’s activation group context.

PGORDENT then did an EXFMT (write/read to display file), waiting for the operator to press a key. The *EXCLRD lock on the ORDMST record was held for the entire workstation wait — typically 3 to 8 minutes during peak processing.

WRKOBJLCK diagnosis and the MCH3601 chain

The retainer consultant ran WRKOBJLCK FILE(ORDLIB/ORDMST) MBR(*FIRST) while an order entry operator had the order status check screen displayed. The output showed exactly one *EXCLRD lock held by interactive job ORDENTRY under user ORDMGR — on the ORDMST record for the order number the operator had just looked up. The lock had been held for 6 minutes.

Meanwhile, job ORDPICK under user WHPICK was executing a COOL:2E-generated picking confirmation function (PGORDPCK), which also used a Change function type on ORDMST. PGORDPCK’s CHAIN ORDNO ORDMST hit the same record already locked by ORDENTRY. WAITRDC(60) in the generated F-spec meant ORDPICK waited 60 seconds, then received an MCH3601 exception. The warehouse order picking queue fell behind and the pickers had to re-scan the same items repeatedly when their terminals locked up.

Fix: Change Function Type to Process in COOL:2E model

The fix required one model change and a regeneration cycle:

  1. In the COOL:2E model APPMODEL (library SYNLIB), open function PGORDCHK
  2. Change Function Type from Change to Process
  3. Regenerate PGORDCHK — COOL:2E now generates FORDMST IF E K DISK (IF = input-only, no lock on CHAIN)
  4. Compile: CRTRPGMOD PGM(ORDLIB/PGORDCHK) SRCFILE(ORDLIB/QRPGLESRC) then CRTPGM PGM(ORDLIB/PGORDCHK)

After the fix: WRKOBJLCK FILE(ORDLIB/ORDMST) showed zero *EXCLRD locks during order entry screen waits. PGORDPCK no longer timed out. MCH3601 errors: eliminated. The consultant also audited all other COOL:2E Change functions that contained no UPDATE statement in their action diagram — found three more mismodeled as Change when Process was correct — and corrected all of them in the same session.

How HourTab helps COOL:2E retainer consultants

COOL:2E retainer work is inherently episodic: a production lock issue escalates, the consultant diagnoses in 1–2 hours, the fix takes another hour, regression testing takes another hour. The distributor’s IT manager asked how much of the retainer budget this incident consumed. The retainer consultant pulled up their HourTab link — 3.5 hours logged with timestamps — and sent it in a reply. No spreadsheet attachment, no PDF invoice, no follow-up email asking “how many hours do we have left.”

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: COOL:2E developer retainers

What does a COOL:2E developer on retainer typically do?

A COOL:2E (Synon/2E / ARCAD Application Lifecycle Manager) developer on monthly retainer covers COOL:2E application model maintenance (function type selection: Add, Change, Delete, Inquire, Process — each generates a different RPG F-spec access mode), access path design (logical files, physical file key structures, join logical files for multi-file retrieval), action diagram logic maintenance (IF/ELSE conditions, WHILE loops, CALL FUNCTION for subprogram invocation, SET RETURN CODE for error signaling, UNLOCK FILE for explicit lock release), generated ILE RPG source review and override (CRTRPGMOD, CRTPGM, CRTSRVPGM compilation after regeneration), display file DDS maintenance (SFLCTL/SFL subfile records, INVITE keyword for workstation read, function key indicators CF01–CF24), data structure and reference field synchronization with DB2 for i physical file DDS, COOL:2E model migration between IBM i OS versions (7.3 to 7.4 to 7.5), and post-regeneration regression testing of generated programs.

What COOL:2E debugging work is most commonly underlogged?

COOL:2E functions modeled with Change type when the actual I/O intent is inquiry-only (Process type) are the most systematically underlogged COOL:2E retainer work. Pattern: PGORDCHK modeled as Change function type; COOL:2E generates FORDMST UF A E K DISK WAITRDC(60) F-spec (update-full — CHAIN acquires *EXCLRD lock); PGORDCHK is called from PGORDENT interactive order entry; PGORDCHK CHAINs ORDNO ORDMST — acquires DB2 for i *EXCLRD record lock; IF ORD-STATUS = ‘PICKED’ → SET RETURN CODE DUPLICATE-ORDER → RETURN (no UNLOCK FORDMST — *EXCLRD held in calling activation group); PGORDENT waits at EXFMT for user input with lock still held; warehouse ORDPICK job tries CHAIN ORDNO ORDMST → WAITRDC(60) timeout → MCH3601 exception; WRKOBJLCK FILE(ORDLIB/ORDMST): ORDENTRY *EXCLRD 1 record; fix: change Function Type from Change to Process in COOL:2E model; regenerate PGORDCHK; CRTRPGMOD/CRTPGM; Process type generates FORDMST IF E K DISK F-spec (input-only — no lock on CHAIN); WRKOBJLCK: 0 *EXCLRD. Log must name the COOL:2E function name, wrong function type, generated F-spec keyword (UF vs IF), CHAIN opcode, lock duration, and WRKOBJLCK before/after.

What are typical COOL:2E developer retainer rates?

Entry-level COOL:2E developers with basic COOL:2E model navigation, generated RPG source reading, and function type identification typically bill at $85 to $140 per hour. Mid-level COOL:2E developers with function type design (Add/Change/Delete/Inquire/Process/Browse distinctions), action diagram logic maintenance (IF/ELSE, WHILE, CALL FUNCTION, SET RETURN CODE), access path design (logical file creation, join logical files), display file DDS subfile maintenance (SFLCTL, SFL, SFLDSP, SFLCLR indicators), post-regeneration IBM i compilation (CRTRPGMOD, CRTPGM, CRTSRVPGM binding directory updates), and COOL:2E-generated record lock diagnosis (WRKOBJLCK, WAITRDC tuning, UNLOCK FILE action insertion) typically bill at $125 to $205 per hour. Senior COOL:2E developers with COOL:2E model repository administration (COOL:2E model backup/restore, cross-reference analysis, WHERE USED for function impact analysis), COOL:2E to native ILE RPG migration planning (extracting generated source and removing COOL:2E regeneration dependency), ARCAD ALM (formerly COOL:2E 6.x) version upgrade support, COOL:2E model documentation generation, and integration of COOL:2E-generated programs with modern IBM i REST APIs typically bill at $185 to $310 per hour. Monthly retainer ranges: $3,400 to $7,000/month for COOL:2E model maintenance, action diagram fixes, and post-regeneration compilation; $6,200 to $12,500/month for active COOL:2E development, model architecture, and modernization planning.

What should a COOL:2E developer retainer agreement include?

A COOL:2E retainer agreement should specify: COOL:2E version (5.x, 6.x) or ARCAD ALM version, and IBM i OS version (7.3, 7.4, 7.5) — regeneration behavior and generated RPG syntax differ by version; whether the retainer covers model changes (action diagram edits, function type corrections, access path additions) and regeneration, or only generated source fixes without model changes; which COOL:2E model library is in scope (APPMODEL in SYNLIB, ORDMODEL in ORDLIB — the model library determines which functions can be regenerated); whether post-regeneration compilation infrastructure is in scope (binding directories, CRTSRVPGM, ILE activation groups); whether COOL:2E display file DDS maintenance is in scope (subfile record format changes, new function key definitions, validity checking keywords); whether generated record lock diagnosis is in scope (WRKOBJLCK for COOL:2E-generated update programs, WAITRDC timeout analysis, UNLOCK FILE action insertion in action diagrams); whether COOL:2E model documentation and WHERE USED cross-reference analysis are in scope; and whether ARCAD ALM product license support contacts or PTF coordination are included in the retainer scope.

How should COOL:2E developer retainer hours be logged?

Log each COOL:2E retainer session with the COOL:2E version, IBM i OS version, function name, function type, model library, generated RPG F-spec, and before/after metric. For Change function type lock: COOL:2E 6.0 IBM i 7.4; model APPMODEL library SYNLIB; function PGORDCHK type CHANGE; COOL:2E generates FORDMST UF A E K DISK WAITRDC(60) (update-full — CHAIN acquires *EXCLRD lock); called from PGORDENT interactive order entry; PGORDENT in job ORDENTRY user ORDMGR; PGORDCHK CHAINs ORDNO=‘ORD-0044891’ ORDMST — *EXCLRD acquired; IF ORD-STATUS=‘PICKED’ SET RETURN CODE DUPLICATE-ORDER RETURN (no UNLOCK ORDMST — *EXCLRD held); PGORDENT EXFMT ORDENTD (waits for user keystroke — lock held during screen wait, up to 8 minutes); warehouse job ORDPICK user WHPICK tries CHAIN ORDNO ORDMST — WAITRDC(60) timeout — MCH3601 exception — ORDPICK job ends abnormally; WRKOBJLCK FILE(ORDLIB/ORDMST) MBR(*FIRST): ORDENTRY *EXCLRD 1 record 8 min; fix: PGORDCHK Function Type changed from CHANGE to PROCESS in COOL:2E model APPMODEL; regenerate PGORDCHK — generates FORDMST IF E K DISK F-spec (input-mode, no lock on CHAIN); CRTRPGMOD PGM(ORDLIB/PGORDCHK) SRCFILE(ORDLIB/QRPGLESRC); CRTPGM PGM(ORDLIB/PGORDCHK); WRKOBJLCK: 0 *EXCLRD during screen wait; hours: 1h WRKOBJLCK diagnosis + 0.5h COOL:2E model change + 0.5h regenerate/compile + 1h regression test.

See also: IBM AS/400 / IBM i developer retainer · Db2 developer retainer · JCL developer retainer