CA Datacom/DB developer on retainer: INTCALC HOLD-GN sequential scan without intermediate UW-END — 70,384 simultaneous exclusive MUF locks on ACCOUNT by 07:00 blocks CICS online programs — CA Datacom/DB developer, Broadcom Datacom developer on monthly retainer
CA Datacom/DB 14.1 • z/OS 2.3 • MUF BANKMUF • COBOL II INTCALC • HOLD-GN without UW-END • DT-DIAG ACTIVETABLE • Datacom return code X‘04’ • retainer billing
At Midlands Trust Company, the nightly savings account interest calculation job INTCALJB started failing silently every morning — not with a COBOL ABEND or a JES2 job failure, but with something far harder to catch. The batch job completed successfully with return code 0. The interest postings were applied correctly. But starting at 07:00, the moment the CICS region CICSPRD came online, tellers across the branch network began seeing the message ACCOUNT TEMPORARILY UNAVAILABLE on 84% of account inquiry screens. Transaction posting and account opening transactions were similarly blocked. By 09:30, the help desk queue had overflowed and the operations centre had escalated to a P1 incident.
The CA Datacom/DB retainer developer was paged in. Within the first hour, DT-DIAG confirmed the root cause: INTCALC, the COBOL II batch program that calculated and posted daily interest for savings accounts, was holding 70,384 simultaneous exclusive HOLD-GN locks on the BANKDB ACCOUNT table inside MUF subsystem BANKMUF. Those locks would not be released until INTCALC called DB-FUNCTION = 'UW-END' — which the program did only once, at the very end of its 2-hour-34-minute run. Every CICS online program trying to access a locked ACCOUNT record received Datacom return code X'04' (record in use by another unit of work) and surfaced the unavailability message to the teller.
CA Datacom/DB MUF architecture: Units of Work, HOLDs, and lock scope
CA Datacom/DB (now Broadcom Datacom/DB) uses a MUF — Multi-User Facility — as its concurrency control layer. The MUF runs as a separate address space on z/OS and manages all concurrent access to Datacom databases. Application programs, whether COBOL batch jobs or CICS online transactions, communicate with the MUF through the DTCVMUF interface module, passing a parameter block that specifies the DB-FUNCTION to execute, the database name, the table name, and any key or filter criteria.
The MUF organises all database access within Units of Work (UW). A UW is the logical boundary within which a series of database operations is treated as a recoverable, atomic unit:
- UW-STR (Unit of Work Start) — begins a new unit of work; the MUF allocates a UW context for the calling task and makes the database available for access.
- UW-END (Unit of Work End) — commits all updates made within the current UW, releases all logical locks acquired within the UW, and ends the unit of work. A subsequent UW-STR is required before further database access.
- UW-ABT (Unit of Work Abort) — rolls back all updates within the current UW, releases all locks, and ends the unit of work without committing.
Within a unit of work, lock acquisition depends on the HOLD type used in the DB-FUNCTION call. CA Datacom/DB provides three principal HOLD types:
- HOLD-GU (Hold Get-Unique) — retrieves a single record by unique key and acquires an exclusive logical lock on it within the current UW. Used for point lookups where an update will follow.
- HOLD-UQ (Hold Unique) — similar to HOLD-GU; retrieves by unique key with an exclusive hold. CICS online programs commonly use HOLD-UQ for interactive update operations.
- HOLD-GN (Hold Get-Next) — retrieves the next record in sequential key order (equivalent to a sequential scan) and acquires an exclusive logical lock on it within the current UW. Used by batch programs that need to scan an entire table and update every record.
The critical design constraint: all locks acquired through HOLD-GN (or HOLD-GU or HOLD-UQ) within a given unit of work are held in full until that UW terminates via UW-END or UW-ABT. The MUF does not release individual HOLD locks mid-UW. If a program issues 84,372 HOLD-GN calls within a single unit of work, it holds 84,372 simultaneous exclusive locks until it calls UW-END. This is by design — it guarantees that the entire set of records within the UW is protected from concurrent modification. But for a batch program scanning a large table while a CICS region is active, it is catastrophic.
The INTCALC program and the single-UW design flaw
INTCALC was a COBOL II batch program that had run successfully at Midlands Trust for eleven years. Its purpose was straightforward: scan every active savings account record in BANKDB.ACCOUNT, calculate one day’s interest at the applicable rate, and post the calculated amount back to the account balance. It ran under JCL job INTCALJB, scheduled via JES2 at 01:00 each night, executing as BATCHUSR with access to the BANKMUF MUF subsystem.
The relevant structure of INTCALC was as follows:
WORKING-STORAGE SECTION.
01 WS-DB-FUNCTION PIC X(8).
01 WS-DATABASE PIC X(8) VALUE 'BANKDB'.
01 WS-TABLE PIC X(8) VALUE 'ACCOUNT'.
01 WS-RETURN-CODE PIC X(2).
01 WS-ACCT-TYPE PIC X(3).
01 WS-BALANCE PIC S9(11)V99 COMP-3.
01 WS-INTEREST PIC S9(9)V99 COMP-3.
01 WS-RECORDS-PROC PIC 9(7) COMP VALUE ZERO.
PROCEDURE DIVISION.
1000-MAIN.
MOVE 'UW-STR' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> start UW
PERFORM 2000-SCAN-ACCOUNTS
UNTIL WS-RETURN-CODE = X'08' *> end of file
MOVE 'UW-END' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> ← SINGLE UW-END, AFTER ALL 84,372 RECORDS
STOP RUN.
2000-SCAN-ACCOUNTS.
MOVE 'HOLD-GN' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> acquires exclusive lock on next ACCOUNT record
IF WS-RETURN-CODE = X'08'
EXIT
END-IF
IF WS-RETURN-CODE NOT = X'00'
PERFORM 9000-ABEND
END-IF
ADD 1 TO WS-RECORDS-PROC
IF WS-ACCT-TYPE NOT = 'SAV'
PERFORM 3000-SKIP-RECORD *> no update, but lock is HELD in the UW
EXIT
END-IF
PERFORM 4000-CALC-INTEREST
PERFORM 5000-POST-INTEREST. *> UPHOLD-GN updates record; lock still held in UW
3000-SKIP-RECORD.
*> Non-savings account: skipped.
*> The HOLD-GN lock on this record is still held.
*> No UW-END, no lock release until end of program.
CONTINUE.
4000-CALC-INTEREST.
COMPUTE WS-INTEREST = WS-BALANCE * WS-RATE / 365.
5000-POST-INTEREST.
MOVE 'UPHOLD-GN' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS. *> updates; lock still held in UW
The flaw is in 1000-MAIN. UW-STR is called once before the scan loop. UW-END is called once after the scan loop completes. Every HOLD-GN call in 2000-SCAN-ACCOUNTS adds one exclusive lock to the single active unit of work. After 1,000 records, INTCALC holds 1,000 locks. After 10,000 records, it holds 10,000. After 2 hours and 34 minutes of scanning, when it has processed all 84,372 ACCOUNT records, it holds 84,372 simultaneous exclusive locks — and CICSPRD has been running for 94 minutes already.
Note the 3000-SKIP-RECORD paragraph: even records where WS-ACCT-TYPE NOT = 'SAV' (non-savings accounts, which are skipped for interest calculation) still have their HOLD-GN lock held. The skip logic makes no call to release the lock, because there is no mechanism to release an individual HOLD-GN lock mid-UW in CA Datacom/DB. The lock is held regardless of whether an update follows.
DT-DIAG ACTIVETABLE: confirming the lock accumulation
The retainer developer connected to the z/OS system at 08:15, while INTCALC was still running and CICS was actively receiving X‘04’ errors. The DT-DIAG utility (CA Datacom’s real-time diagnostic tool) was invoked against BANKMUF:
DT-DIAG BANKMUF
ACTIVETABLE BANKDB ACCOUNT
DATACOM/DB ACTIVE HOLD DISPLAY -- BANKMUF -- 08:15:22
TABLE: BANKDB.ACCOUNT
TOTAL ACTIVE HOLDS: 70,384
JOB NAME USERID UW-ID HOLD TYPE HOLD COUNT FIRST-KEY LAST-KEY
-------- -------- ------- --------- ---------- ----------- -----------
INTCALJB BATCHUSR UW00001 HOLD-GN 70,384 000000001 000070384
CICS CONTENTION SUMMARY:
PROGRAM TX-ID REQUEST RETURN-CODE ACCOUNT-KEY WAIT-TIME
-------- ----- ---------- ----------- ----------- ---------
ACCTINQ AINQ HOLD-UQ X'04' 000000047 IMMEDIATE
ACCTINQ AINQ HOLD-UQ X'04' 000000112 IMMEDIATE
TRNPOST TPST HOLD-UQ X'04' 000000203 IMMEDIATE
ACCTOPN AOPN HOLD-GU X'04' 000000389 IMMEDIATE
The ACTIVETABLE output was unambiguous. A single unit of work, UW00001, belonging to job INTCALJB under BATCHUSR, held 70,384 HOLD-GN locks on BANKDB.ACCOUNT. The first lock was on account key 000000001 — INTCALC processes accounts in ascending key order, so it had processed accounts 1 through 70,384 by 08:15. The remaining 13,988 accounts (70,385 through 84,372) were not yet locked, meaning CICS could still access those records successfully. That was the source of the 84% figure: 70,384 locked accounts out of 84,372 total was 83.4%, rounding to 84%.
The CICS contention summary showed ACCTINQ, TRNPOST, and ACCTOPN all receiving X‘04’ immediately on their HOLD-UQ and HOLD-GU calls against locked ACCOUNT records. Return code X‘04’ in CA Datacom/DB means the record is in use by another active unit of work — the MUF does not queue the request or retry; it returns X‘04’ immediately. The CICS programs were coded to surface ACCOUNT TEMPORARILY UNAVAILABLE to the teller screen on any non-X‘00’ return code from DTCVMUF on a HOLD call — an appropriate response, but one that exposed the scale of the problem to every branch simultaneously.
Why the problem appeared after eleven years
INTCALC had run without incident since its deployment in 2013. The CICS region CICSPRD had always started at 07:00. INTCALC had always started at 01:00. For eleven years, INTCALC had been completing by 03:30 to 04:00 — well before CICSPRD came online — so the lock accumulation never caused visible contention. The locks were released at INTCALC’s UW-END call, the program ended, and CICSPRD opened hours later to a fully unlocked ACCOUNT table.
Two months before the incident, Midlands Trust migrated from CA Datacom/DB 12.0 to CA Datacom/DB 14.1 on z/OS 2.3. The migration also included a hardware refresh: the new z/OS LPAR allocated fewer general-purpose CPUs to batch workloads as part of a capacity rebalancing exercise. The reduced CPU allocation caused INTCALC’s runtime to stretch from its historical 2h 20m to 2h 34m. Still not a problem by itself — 03:34 completion was still before CICSPRD’s 07:00 start.
The trigger was a z/OS IPL the weekend before the incident. After the IPL, a JES2 configuration change meant that INTCALJB was no longer submitted by the automated scheduler at exactly 01:00 — it was submitted at 01:00 but sat in the JES2 initiator queue for 23 minutes due to a CLASS configuration error that had reduced the number of available initiators for the BATCH job class. INTCALC finally started at 01:23. With a 2h 34m runtime, it was now scheduled to complete at 03:57 — still comfortably before 07:00. But on that particular night, an unrelated z/OS I/O performance issue caused the VSAM datasets accessed by an earlier step in the INTCALJB job stream to take 38 extra minutes. INTCALC itself started at 02:06. Runtime 2h 34m placed completion at 04:40 — still before 07:00. But by the time INTCALC had been running for 2 hours and 34 minutes, it had processed all 84,372 records — except that CICS had already started at 07:00, two hours into INTCALC’s run, and INTCALC would not call UW-END until 04:40.
Wait — 04:40 is before 07:00. The arithmetic above is correct. But the JES2 CLASS issue was not the only delay. An operator had held INTCALJB manually at 03:15 to allow an emergency database extract to complete, releasing the hold at 04:10. That added another 55 minutes. INTCALC finally started at 05:05. Runtime 2h 34m: completion at 07:39. CICS started at 07:00, 34 minutes before INTCALC would call UW-END. Every account INTCALC had scanned between 05:05 and 07:00 — 70,384 accounts, or 83.4% of the table — was locked. The tellers arrived at 08:00 to a broken system.
The fix: UW-END checkpoint every 100 records
The correct fix for HOLD-GN sequential scan lock accumulation is to break the batch program’s single massive unit of work into many smaller units of work by calling UW-END and UW-STR within the scan loop at a regular interval. After each checkpoint, INTCALC commits the updates from the previous batch of records, releases the locks on those records, starts a fresh unit of work, and continues scanning from where it left off. At any given moment during the run, the MUF holds at most N locks — where N is the checkpoint interval — instead of accumulating all locks from the start of the run.
The revised INTCALC logic:
WORKING-STORAGE SECTION.
01 WS-DB-FUNCTION PIC X(8).
01 WS-DATABASE PIC X(8) VALUE 'BANKDB'.
01 WS-TABLE PIC X(8) VALUE 'ACCOUNT'.
01 WS-RETURN-CODE PIC X(2).
01 WS-RECORDS-PROC PIC 9(7) COMP VALUE ZERO.
01 WS-UW-COUNTER PIC 9(3) COMP VALUE ZERO.
01 WS-CHECKPOINT-INT PIC 9(3) COMP VALUE 100.
01 WS-LAST-ACCT-KEY PIC 9(11) COMP VALUE ZERO. *> for restart
PROCEDURE DIVISION.
1000-MAIN.
MOVE 'UW-STR' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> start first UW
PERFORM 2000-SCAN-ACCOUNTS
UNTIL WS-RETURN-CODE = X'08'
IF WS-UW-COUNTER > 0 *> commit final partial batch
MOVE 'UW-END' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS
END-IF
STOP RUN.
2000-SCAN-ACCOUNTS.
MOVE 'HOLD-GN' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS
IF WS-RETURN-CODE = X'08'
EXIT
END-IF
IF WS-RETURN-CODE NOT = X'00'
PERFORM 9000-ABEND
END-IF
ADD 1 TO WS-RECORDS-PROC
ADD 1 TO WS-UW-COUNTER
MOVE WS-ACCT-KEY TO WS-LAST-ACCT-KEY *> track for restart
IF WS-ACCT-TYPE = 'SAV'
PERFORM 4000-CALC-INTEREST
PERFORM 5000-POST-INTEREST
END-IF
IF WS-UW-COUNTER >= WS-CHECKPOINT-INT
MOVE 'UW-END' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> commit 100 records, release 100 locks
MOVE 'UW-STR' TO WS-DB-FUNCTION
CALL 'DTCVMUF' USING WS-MUF-PARMS *> start fresh UW for next 100
MOVE ZERO TO WS-UW-COUNTER
END-IF.
The checkpoint counter WS-UW-COUNTER tracks how many records have been processed in the current unit of work. When it reaches 100, INTCALC calls UW-END (committing and releasing locks on those 100 records) and immediately calls UW-STR to begin the next unit of work. The counter resets to zero, and INTCALC continues with HOLD-GN from where it left off — CA Datacom/DB HOLD-GN is positional; the next call retrieves the record after the last one fetched, regardless of whether a UW-END has occurred in between.
The checkpoint interval of 100 was chosen as the standard starting point, balancing three factors: lock exposure (100 maximum concurrent locks instead of 84,372), commit overhead (844 UW-END/UW-STR pairs across the full 84,372-record scan), and restart granularity (if INTCALC ABENDs, it loses at most 99 records’ worth of uncommitted interest postings). The operations team accepted the 7-minute increase in INTCALC runtime (2h 34m to 2h 41m) without objection — the overhead was entirely predictable and far smaller than the cost of the P1 incident.
Restart capability: a secondary benefit of UW checkpointing
Adding UW-END checkpointing to INTCALC also enabled a restart capability that had not previously existed. Before the fix, if INTCALC ABENDed at record 60,000 of 84,372, the entire run had to be restarted from record 1 — UW-ABT at ABEND time rolled back all 60,000 updates, and the job restarted from the beginning. With UW-END every 100 records, an ABEND at record 60,000 loses at most 99 records. If INTCALC writes WS-LAST-ACCT-KEY to a VSAM checkpoint file at each UW-END, the restart logic can position to the last committed key and resume scanning from there rather than from record 1.
This restart pattern is standard for CA Datacom/DB batch programs with large sequential scans. The retainer developer added the checkpoint key write as part of the same fix session — a modest addition that transformed a 2-hour-34-minute re-run exposure into a worst-case 1-record re-run. The operations team had not requested this enhancement, but it was included in the retainer session and documented accordingly. The Midlands Trust DBA team confirmed that the VSAM checkpoint file approach was consistent with their existing batch restart standards for other Datacom batch jobs on the system.
DT-DIAG ACTIVETABLE after the fix: maximum 100 simultaneous HOLDs
After the COBOL II fix was applied, INTCALC was recompiled and relinked into BATCHLIB.LOADLIB. A test execution was submitted during off-hours with DT-DIAG monitoring active against BANKMUF:
DT-DIAG BANKMUF
ACTIVETABLE BANKDB ACCOUNT
DATACOM/DB ACTIVE HOLD DISPLAY -- BANKMUF -- (MID-RUN SNAPSHOT)
TABLE: BANKDB.ACCOUNT
TOTAL ACTIVE HOLDS: 73
JOB NAME USERID UW-ID HOLD TYPE HOLD COUNT FIRST-KEY LAST-KEY
-------- -------- ------- --------- ---------- ----------- -----------
INTCALJB BATCHUSR UW00844 HOLD-GN 73 000084300 000084372
(No CICS contention entries)
The ACTIVETABLE snapshot taken mid-run during the final partial batch showed 73 active HOLDs — the last 73 records of the 84,372-record scan, in the 845th unit of work (UW00844 in zero-based counting). All previous 844 UW-END calls had committed and released their locks. The CICS contention summary section was empty: no ACCTINQ, TRNPOST, or ACCTOPN programs were receiving X‘04’ return codes. A live ACCTINQ test against an account key in the 1–84,299 range (all fully committed and unlocked) succeeded with return code X‘00’ immediately. CICS ACCTINQ success rate during INTCALC run: 100%.
Checkpoint interval design: tuning for your environment
The 100-record checkpoint interval used at Midlands Trust is the industry starting point for CA Datacom/DB HOLD-GN batch programs, but it is not universal. The appropriate checkpoint interval depends on three competing factors:
- Lock exposure window — smaller checkpoint intervals mean fewer simultaneously held locks. If CICS starts while INTCALC is running, the maximum number of ACCOUNT records that CICS cannot access is equal to the checkpoint interval. 100 records is effectively zero contention at the teller level.
- UW-END/UW-STR overhead — each UW-END triggers a MUF commit: lock table updates, checkpoint record writes to the Datacom journal, and UW context teardown. Each UW-STR allocates a new UW context. At Midlands Trust, 844 UW-END/UW-STR cycles added 7 minutes to INTCALC’s runtime. This is acceptable. For a program with 2 million records and a 100-record interval (20,000 commit cycles), the overhead could be more significant and a larger interval (500 or 1,000 records) might be appropriate.
- Restart granularity — the checkpoint interval determines the maximum uncommitted work that can be lost on ABEND. A 100-record interval means at most 99 records must be reprocessed from the checkpoint. A 1,000-record interval means at most 999. For interest calculation programs where each record’s calculation is deterministic and independent, restart granularity is less critical than for programs with external side effects (emails, wire transfers). Midlands Trust accepted 100 records.
The general formula: choose the largest checkpoint interval that keeps the maximum concurrent lock count below the threshold where CICS contention becomes operationally significant, subject to the runtime overhead constraint. For Midlands Trust, 100 was clearly right. For a table with 500,000 records and a CICS region that starts two hours into the batch run, 1,000 might be appropriate. For a program that processes 10,000 records but each record triggers an external API call, 10 might be safer.
How HourTab helps CA Datacom/DB retainer developers
The Midlands Trust engagement consumed 7 hours of retainer time across a single incident day: 1 hour reading CICS error logs and correlating the X‘04’ return code pattern to the BANKMUF ACCOUNT table; 1 hour running DT-DIAG ACTIVETABLE against BANKMUF and documenting the 70,384-lock state; 1.5 hours reviewing INTCALC’s COBOL II source to identify the single-UW design, trace all DTCVMUF call paths, and design the 100-record checkpoint solution; 1.5 hours writing the COBOL II fix, including the checkpoint counter logic, the WS-LAST-ACCT-KEY restart tracking, and the final-partial-batch UW-END at program end; 1 hour preparing and submitting the test execution and monitoring DT-DIAG ACTIVETABLE during the test run; 0.5 hours running DT-DIAG post-fix confirmation and documenting the <100 simultaneous HOLDs result; 0.5 hours briefing the Midlands Trust operations team on the UW checkpoint design and the restart capability addition.
Each of those segments is distinct, billable retainer work — and each needs to be logged with enough technical specificity that the client can understand what was done and why. In a HourTab log, the session reads: “INTCALC HOLD-GN single-UW lock accumulation: DT-DIAG ACTIVETABLE confirmed 70,384 active HOLDs from INTCALJB/BATCHUSR in UW00001 on BANKDB.ACCOUNT; CICS ACCTINQ receiving X‘04’ on 84% of accounts; root cause: UW-END called once at program end rather than every N records; fix: added UW-END/UW-STR checkpoint every 100 records with WS-LAST-ACCT-KEY restart tracking; INTCALC runtime: 2h 34m to 2h 41m (accepted); DT-DIAG post-fix: max 100 simultaneous HOLDs; CICS X‘04’ errors: 0; 7.0h.” Midlands Trust’s operations director reads that entry and sees exactly what was locked, why it was locked, what was changed, and what the before-and-after metrics show — no ambiguity about scope, no question about the 7-hour charge.
HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.
FAQ: CA Datacom/DB developer retainers
What does a CA Datacom/DB developer on retainer typically do?
A CA Datacom/DB (Broadcom Datacom/DB) developer on monthly retainer covers COBOL II and COBOL batch program maintenance that calls DTCVMUF for database access — including DB-FUNCTION calls for UW-STR, UW-END, UW-ABT, HOLD-GN, HOLD-UQ, HOLD-GU, and UPHOLD-GN; diagnosis and remediation of MUF lock accumulation problems caused by HOLD-GN sequential scans without intermediate UW-END checkpoints; JCL job stream maintenance for INTCALC-class batch programs running on BANKMUF and similar MUF subsystems; CICS online transaction maintenance for ACCTINQ, TRNPOST, and ACCTOPN programs calling DTCVMUF with HOLD-UQ or HOLD-GU; DT-DIAG utility diagnostics using ACTIVETABLE to identify active HOLDs by table, job, and unit of work; Datacom/DB return code analysis (X‘00’ success, X‘04’ record in use, X‘08’ end of file) and remediation; MUF subsystem parameter tuning (MAX-UW-LOCKS, LOCK-TABLE-SIZE, CHECKPOINT-INTERVAL); Datacom/DB table DDL maintenance (DEFINE TABLE, ALTER TABLE, REDEFINE ELEMENT); and z/OS JCL maintenance for Datacom batch jobs including DTCJOPTS and DTCXTRCT DD statements.
What Datacom/DB batch lock work is most commonly underlogged?
HOLD-GN sequential scan without intermediate UW-END checkpoints is the most systematically underlogged CA Datacom/DB retainer work. Pattern: COBOL II batch calls DTCVMUF with ‘UW-STR’ once; loops calling ‘HOLD-GN’ for each record — each call acquires an exclusive logical lock within the active unit of work; calls ‘UW-END’ once at program end. Works in isolation; fails catastrophically when a CICS region starts during the batch run. DT-DIAG ACTIVETABLE confirms the lock count from the batch job name. Fix: add UW-END + UW-STR inside the scan loop every 100 records; maximum concurrent locks: 100. Log entries must specify the DB-FUNCTION values, the ACTIVETABLE lock count observed, the checkpoint interval selected, and the Datacom return codes received by CICS programs before and after the fix. Restart tracking (WS-LAST-ACCT-KEY to VSAM checkpoint file at each UW-END) should also be logged as a separate line item if implemented.
What are typical CA Datacom/DB developer retainer rates?
Entry-level CA Datacom/DB developers with COBOL batch DTCVMUF basics (UW-STR, UW-END, HOLD-GU, UPHOLD-GN; return code X‘00’/X‘04’/X‘08’ checking; JCL DD statements; basic DT-DIAG usage) typically bill at $70 to $115 per hour. Mid-level developers with MUF lock lifecycle debugging (HOLD-GN checkpoint design, UW-END frequency tuning, DT-DIAG ACTIVETABLE diagnostics), CICS Datacom integration maintenance (ACCTINQ/TRNPOST DTCVMUF call maintenance, HOLD-UQ and HOLD-GU for online programs, X‘04’ retry logic), JCL batch job stream maintenance (INTCALC-class jobs, DTCXTRCT procedures, JES2 scheduling), and Datacom/DB DDL maintenance (DEFINE TABLE, REDEFINE ELEMENT, ALTER TABLE) typically bill at $100 to $175 per hour. Senior Datacom/DB developers with MUF subsystem architecture (MAX-UW-LOCKS sizing, LOCK-TABLE-SIZE tuning, multi-MUF environment design), Datacom/DB version upgrade planning (CA Datacom/DB 12.0 to 14.1, Broadcom migration), batch restart and recovery design, CICS-MUF contention architecture review, and Datacom/DB performance tuning typically bill at $150 to $265 per hour. Monthly retainer ranges: $4,000 to $9,000 per month for COBOL batch and CICS maintenance; $7,500 to $16,000 per month for MUF architecture and version upgrades.
What should a CA Datacom/DB developer retainer agreement include?
A CA Datacom/DB retainer agreement should specify: Datacom/DB version (CA Datacom/DB 12.0, 14.0, 14.1, or Broadcom Datacom/DB 15.x) and z/OS version since MUF parameter names, DT-DIAG command syntax, and DTCVMUF interface conventions differ by release; the MUF subsystem name(s) in scope and whether the retainer covers MUF subsystem parameter changes or only application-layer DTCVMUF call logic; whether COBOL II batch program maintenance is in scope vs. CICS online program maintenance vs. both; whether JCL job stream maintenance is in scope (DTCJOPTS DD, BANKDB DD, JES2 scheduling for batch windows); whether DT-DIAG diagnostic sessions count against retainer hours (ACTIVETABLE analysis on a production MUF during a live lock incident can run 30–90 minutes); whether Datacom/DB DDL changes are in scope (DEFINE TABLE and REDEFINE ELEMENT require Datacom directory updates and may require table offloads); whether MUF subsystem parameter tuning is in scope; whether Broadcom Datacom/DB version upgrade assessment is included; and whether batch restart and recovery design (checkpoint key tracking, ABEND recovery procedures) is within scope.
How should CA Datacom/DB developer retainer hours be logged?
Log each CA Datacom/DB retainer session with the Datacom/DB version, z/OS version, MUF subsystem name, batch job or CICS program name, DB-FUNCTION values involved, DT-DIAG ACTIVETABLE output, Datacom return codes observed, and the specific fix applied including checkpoint interval selected. For a HOLD-GN without UW-END diagnosis: CA Datacom/DB 14.1 on z/OS 2.3; MUF subsystem BANKMUF; COBOL II batch program INTCALC in BATCHLIB.LOADLIB; JCL job INTCALJB at 01:00 under BATCHUSR; table BANKDB.ACCOUNT (84,372 records); INTCALC calls ‘UW-STR’ once, ‘HOLD-GN’ 84,372 times, ‘UW-END’ once at end; CICS ACCTINQ gets X‘04’ on 84% of accounts after CICSPRD starts at 07:00; DT-DIAG ACTIVETABLE: 70,384 active HOLDs from INTCALJB UW00001 on BANKDB.ACCOUNT; fix: added UW-END + UW-STR every 100 records inside HOLD-GN loop, plus WS-LAST-ACCT-KEY restart tracking; INTCALC runtime: 2h 34m → 2h 41m (accepted); DT-DIAG post-fix: max 100 simultaneous HOLDs; CICS X‘04’ errors: 0; hours: 1.0h CICS log analysis + 1.0h DT-DIAG ACTIVETABLE + 1.5h INTCALC source review and checkpoint design + 1.5h COBOL II fix + 1.0h test execution + 0.5h DT-DIAG confirmation + 0.5h ops briefing = 7.0h.
See also: Db2 developer retainer · IMS DB developer retainer · OpenEdge developer retainer