Blog › ICP guides

Easytrieve developer on retainer: PRMVLD01 FILE BILLMST VSAM KSDS UPDATE opens cluster with exclusive write access — CICS billing transactions receive NOTOPEN for 2 hours 37 minutes — CA Easytrieve developer, Broadcom Easytrieve Plus developer on monthly retainer

October 10, 2026 · ~15 min read

A CA Easytrieve Plus developer was maintaining the premium validation batch suite for Keystone Benefits Group, a self-insured employer benefits administrator running on IBM z/OS 2.5 with CICS TS 6.1. The company processed monthly premium billing for 14,000 covered employees across 87 employer groups, and the billing system had two layers: an online CICS transaction layer (used by billing analysts to look up premium status and post manual adjustments) and a nightly batch layer (Easytrieve Plus programs that processed mass premium updates from payroll feeds). The premium billing master file was a VSAM KSDS (Key-Sequenced Data Set) cluster named KEYSTONE.BILLING.MASTER, opened by CICS as file BILLMST at CICS startup and accessed throughout the business day for online billing inquiries and payment posting. Every month, on the 28th at 2:00 AM, the Easytrieve Plus batch program PRMVLD01 (Premium Validation and Posting) ran to validate and post the monthly premium adjustments from the payroll feed into BILLMST.

On the 28th of the month following a 24/7 operations transition, the billing team’s overnight supervisor opened a help desk ticket at 4:37 AM: CICS transaction BVLD (Billing Validation Display) was returning a system error. By 5:15 AM, eight more tickets had been opened: CICS transactions PMNT (Premium Payment Posting), BVLD, and BGRO (Benefits Group Overview) were all failing when they attempted to access BILLMST. The CICS error was EIBRESP=NOTOPEN (EIBRESP code 19 in CICS TS 6.1 — the file is closed and cannot be opened by the transaction). The CICS operations analyst checked CEDA INQUIRE FILE(BILLMST): the BILLMST file showed status CLOSED/DISABLED. CICS had automatically closed BILLMST after an internal VSAM OPEN error. PRMVLD01 had been running since 2:00 AM and was not expected to finish until 4:38 AM based on its normal runtime of 2 hours 37 minutes.

The Easytrieve Plus source for PRMVLD01 began with the following FILE statement: FILE  BILLMST  VSAM  KSDS  UPDATE. In CA Easytrieve Plus, the FILE statement is the declaration that names the file, specifies its storage organization (VSAM, QSAM, KSDS, ESDS, RRDS, or blank for sequential), and controls the access mode that Easytrieve uses when opening the file. The UPDATE keyword in the FILE statement is a critical attribute: it tells Easytrieve Plus to open the VSAM file with write-capable (output) access — specifically, it causes Easytrieve to open the VSAM ACB (Access Method Control Block) with MACRF parameters that include the OUT (output) capability, enabling the program to issue PUT statements to update or add records. The alternative is INPUT (or omitting the attribute, which defaults to INPUT for VSAM files in most Easytrieve Plus configurations), which opens the VSAM ACB with read-only access.

The consequence of opening a VSAM KSDS with write-capable (UPDATE) access is governed by the VSAM cluster’s SHAREOPTIONS attribute, which is set at cluster creation time and defines the rules for concurrent access across jobs and regions. KEYSTONE.BILLING.MASTER had been created with SHAREOPTIONS(1 3) — the default VSAM SHAREOPTIONS for many legacy mainframe installations. VSAM SHAREOPTIONS(1 3) means: cross-region share option 1 (the first number) — only one address space (region) can have the file open for write (output) access at a time; if one region holds the file open with write capability, another region cannot open it at all until the first region closes it. Cross-system share option 3 (the second number) means no sharing across z/OS images in a Sysplex. Under SHAREOPTIONS(1 3), the rule for PRMVLD01 opening BILLMST with UPDATE (write-capable) access from a batch region was stark: as long as PRMVLD01’s batch job held BILLMST open, no other address space could open BILLMST. Including CICS.

In z/OS, when CICS opens a VSAM file at CICS startup (via CEDA DEFINE FILE or CICS CSD OPEN(YES) attribute), CICS holds the VSAM ACB for that file open for the entire duration of the CICS region. CICS opens BILLMST at startup with a shared ACB (read-write for CICS’s own transaction-level operations, but CICS manages VSAM access internally through CICS file control, not through direct multi-region VSAM sharing). PRMVLD01’s batch job, running in a separate address space (batch region), attempted to open BILLMST with UPDATE access. With SHAREOPTIONS(1 3) and CICS holding BILLMST open, there were two possible failure modes depending on timing: if CICS had BILLMST open when PRMVLD01 started, PRMVLD01 would receive a VSAM OPEN error RTNCODE=X’08’ FDBK=X’A4’ (open failed due to incompatible share options — another region holds the file open with a conflicting access mode) and PRMVLD01 would ABEND. If PRMVLD01 opened BILLMST first (before CICS had reopened it after a scheduled file close), CICS’s next attempt to open BILLMST would fail, and CICS would mark BILLMST as CLOSED/DISABLED. In the current incident, the 24/7 operations transition meant CICS was running continuously — PRMVLD01 started at 2:00 AM and opened BILLMST with UPDATE access, causing CICS’s BILLMST ACB to experience a VSAM re-open conflict, leading CICS to close and disable BILLMST automatically.

CA Easytrieve Plus in 2026: the mainframe reporting and data management workhorse of insurance and government

CA Easytrieve was originally developed by Pansophic Systems in the 1970s as a 4GL (fourth-generation language) for IBM mainframe report writing and data extraction — a higher-level alternative to COBOL for the common task of reading files, selecting records, performing calculations, and generating formatted reports. Pansophic released Easytrieve in 1978; Computer Associates (CA) acquired Pansophic in 1991 and continued developing the product as CA Easytrieve Plus. CA Technologies (the renamed Computer Associates) was acquired by Broadcom in 2018, which renamed the product Broadcom Easytrieve Plus, though CA Easytrieve Plus 11 remains the product version designation in the current Broadcom product catalog. Easytrieve is installed at a large fraction of the major US insurance carriers, large health insurance organizations (as claims and premium data extraction language), state and federal government agencies (as a data manipulation language for benefits data in VSAM and Db2 stores), and large banks (as a reporting language for account data extracts). An Easytrieve developer on monthly retainer maintains the batch programs that extract, validate, and format data from the VSAM files and Db2 tables that store the organization’s core business data — premium billing masters, claims history files, account ledgers, and benefits enrollment tables.

The CA Easytrieve Plus FILE statement controls much of the critical behavior of an Easytrieve program’s VSAM interaction. The full FILE statement syntax for a VSAM KSDS is: FILE name VSAM KSDS [INPUT|UPDATE] [STRNO(n)] [BUFSP(n)] [KEYLEN(n)]. The attributes that most commonly create production incidents:

INPUT vs. UPDATE: INPUT opens the VSAM file read-only (Easytrieve issues VSAM GETs but no PUTs). UPDATE opens with write capability (GETs and PUTs). The critical distinction is that INPUT-mode opens are compatible with CICS’s ACB on the same VSAM KSDS under SHAREOPTIONS(1 3) because both Easytrieve (batch) and CICS are reading — not both writing. Under SHAREOPTIONS(1 3), multiple readers can hold a file open concurrently; only the introduction of a writer (UPDATE open) creates the exclusion that prevents other opens.

STRNO (string number): specifies how many VSAM strings (concurrent access paths) to open. STRNO(1) (Easytrieve’s default for sequential processing) opens one string. For sequential report programs that read BILLMST from beginning to end, one string is sufficient. For random access programs (using Easytrieve POINT/GET by key), STRNO affects parallelism.

BUFSP (buffer space): specifies the VSAM buffer space in bytes. For a large VSAM KSDS like BILLMST (4.2 million records, 512 bytes per record — approximately 2GB), the default buffer space (typically one control interval per string) causes single-track I/O. Setting BUFSP to 2MB or 4MB enables multi-track sequential buffering and can reduce PRMVLD01’s runtime by 30–45% for the sequential read of 4.2 million records.

Easytrieve VSAM access diagnosis: IDCAMS LISTCAT, CICS CEDA INQUIRE, and z/OS job log VSAM OPEN errors

The Easytrieve developer’s diagnostic sequence for the BILLMST CICS outage:

Step 1: IDCAMS LISTCAT. The developer ran IDCAMS LISTCAT CLUSTER(KEYSTONE.BILLING.MASTER) ALL to display the VSAM cluster’s catalog entry. The SHAREOPTIONS attribute confirmed SHROPTNS(1 3) — cross-region share option 1, cross-system share option 3. This immediately identified the root structural cause: the VSAM cluster was defined with exclusive writer semantics that prevented concurrent CICS and batch write access.

Step 2: Review PRMVLD01 source. The developer opened the PRMVLD01 source in ISPF and searched for all occurrences of PUT  BILLMST. The search returned zero results. PRMVLD01 had no PUT statements for BILLMST — the program read BILLMST records sequentially (implicit GET in the JOB INPUT section), validated premium amounts, computed adjustment totals, and generated a SYSPRINT output report. It never wrote to BILLMST. The UPDATE attribute in the FILE statement was speculative: a prior developer had written it thinking PRMVLD01 might “need to update records someday,” but the program had been strictly read-only since its creation in 2019.

Step 3: CICS CEDA INQUIRE FILE(BILLMST). This confirmed BILLMST status as CLOSED/DISABLED. The CICS file had been automatically disabled when CICS encountered the VSAM OPEN conflict during PRMVLD01’s batch window. CICS does not automatically re-enable a VSAM file that has been disabled due to a VSAM OPEN error; it remains CLOSED/DISABLED until an operator or administrator issues CEMT SET FILE(BILLMST) OPEN or CEDA INSTALL to re-enable it.

Step 4: z/OS job log VSAM OPEN messages. The z/OS SYSLOG for the batch region showed IEC161I messages (VSAM OPEN warning) associated with the PRMVLD01 job. In the CICS CSMT log (CICS system message transient data queue), the developer found DFHFC0002 messages (CICS file control — VSAM OPEN for file BILLMST failed with VSAM return code). The DFHFC0002 message confirmed the timing: CICS tried to perform a VSAM re-open for BILLMST during the PRMVLD01 batch window and received the VSAM OPEN error due to the exclusive conflict.

The fix was a single-word change in PRMVLD01’s FILE statement:

Before: FILE  BILLMST  VSAM  KSDS  UPDATE
After: FILE  BILLMST  VSAM  KSDS  INPUT

With INPUT, Easytrieve opens BILLMST with a read-only VSAM ACB. Under SHAREOPTIONS(1 3), a read-only open from the batch region is fully compatible with CICS’s concurrent ACB on BILLMST. After the change, PRMVLD01 ran concurrently with CICS for the full 2 hours 37 minutes without any VSAM conflict. BILLMST remained OPEN/ENABLED in CICS throughout the batch run. CICS transactions BVLD, PMNT, and BGRO operated normally. The 847 NOTOPEN errors during the batch window: zero.

As a medium-term improvement, the Easytrieve developer also submitted a VSAM cluster upgrade plan: changing BILLMST from SHAREOPTIONS(1 3) to SHAREOPTIONS(4 4) and enabling VSAM RLS (Record Level Sharing) on the cluster. Under SHAREOPTIONS(4 4) + VSAM RLS, true record-level locking is managed by the CF (Coupling Facility) lock table, allowing multiple regions (batch AND CICS) to both read and update the same VSAM cluster concurrently without dataset-level exclusion. This would be the correct architecture for future Easytrieve programs that actually need to PUT records to BILLMST (e.g., a future PRMPST01 program that posts adjustments directly to BILLMST rather than generating a downstream update file).

Typical CA Easytrieve developer retainer work and what it looks like in a work log

Easytrieve FILE VSAM UPDATE blocking CICS access under SHAREOPTIONS(1 3) is the canonical invisible Easytrieve batch/online conflict for insurance companies, banks, and government agencies that extended CICS online operations to 24/7 coverage while continuing to run Easytrieve batch programs that were originally written for overnight-only windows. It appears in Easytrieve retainer work logs for health insurance premium billing programs (FILE BILLMST VSAM KSDS UPDATE specifying write access never used), bank account extract programs (FILE ACCTMST VSAM KSDS UPDATE for a read-only audit trail extract), government benefits eligibility programs (FILE ELIGMST VSAM ESDS UPDATE for a sequential read of an ESDS file that never writes), and insurance claims history reports (FILE CLAIMHST VSAM KSDS UPDATE for a monthly claims summary extract). The work log entry: “CA Easytrieve Plus 11; program PRMVLD01; EXEC PGM=EZTPA00 JCL step; SYSIN FILE BILLMST VSAM KSDS UPDATE — Easytrieve opens KEYSTONE.BILLING.MASTER VSAM KSDS with write-capable MACRF (OUT included); SHAREOPTIONS(1 3) — cross-region exclusive write; CICS CICSPRD01 has BILLMST ACB open at startup; 2:00 AM PRMVLD01 OPEN triggers VSAM OPEN conflict; CICS BILLMST CLOSED/DISABLED; CICS BVLD/PMNT/BGRO EIBRESP=NOTOPEN; 847 CICS transactions failed 2:00–4:37 AM (2h 37m); CEDA INQUIRE FILE(BILLMST): CLOSED DISABLED confirmed; IDCAMS LISTCAT CLUSTER(KEYSTONE.BILLING.MASTER) ALL: SHROPTNS(1-3) confirmed; PRMVLD01 source: FILE BILLMST VSAM KSDS UPDATE — zero PUT BILLMST in source; fix: FILE BILLMST VSAM KSDS UPDATE changed to FILE BILLMST VSAM KSDS INPUT; CICS NOTOPEN rate: 847/month → 0; medium-term: BILLMST VSAM cluster upgrade plan for SHAREOPTIONS(4 4) + VSAM RLS for future write programs; also BUFSP(4194304) added to FILE BILLMST for 4MB sequential buffer (estimated 35% runtime reduction for 4.2M record sequential read); hours: 2h diagnosis + 0.5h fix + 0.5h test + 1h SHAREOPTIONS upgrade plan.”

Easytrieve Plus DBMS DB2 interface Db2 cursor accumulating row locks without periodic COMMIT is the Easytrieve-specific version of the Db2 batch cursor lock story, relevant for Easytrieve programs that access DB2 tables via the Easytrieve DBMS DB2 clause. An Easytrieve DBMS DB2 program with a TABLE FILE access for a large Db2 table (e.g., PREM_BILLING with 4.2 million rows) that uses the EASYTRIEVE default isolation level of repeatable read (RR) holds Db2 shared (S) locks on all fetched rows for the duration of the cursor. If the Easytrieve program runs for 3 hours and holds RR shared locks on 4.2 million PREM_BILLING rows, online CICS programs that try to UPDATE a row in PREM_BILLING for a specific POLICY_NBR receive Db2 lock timeout SQLCODE -911 after IRLMRWT seconds. Work log: “CA Easytrieve Plus 11; program PRMBILRPT; DBMS DB2; TABLE PREM_BILLING DBMS-TBL1 VSAM (Easytrieve DB2 table FILE alias); default isolation: RR (Repeatable Read); Easytrieve sequential TABLE access to PREM_BILLING (4.2M rows); Db2 S locks on all 4.2M rows held 3h 12m (Easytrieve holds DB2 cursor open for full sequential scan duration); CICS BVLD UPDATE PREM_BILLING for POLICY_NBR LIFE-2024-47821 blocked by RR shared lock; SQLCODE -911 after 60s IRLMRWT; 23 SQLCODE -911 errors per PRMBILRPT run; EZTPA00 system parameter ISOLATION changed from default RR to CS (Cursor Stability — releases S lock on prior row after advancing cursor); CICS lock wait: 23/run → 0; hours: 2h Easytrieve DB2 interface research + 0.5h EZTPA00 ISOLATION parameter change.”

Easytrieve MATCH facility key synchronization error after VSAM KSDS REORG changes key ordering is the most common Easytrieve output error following a VSAM KSDS reorganization. Easytrieve’s MATCH facility processes two or more sequentially sorted input files and produces matched/unmatched output based on key field comparison — conceptually a merge-join on sorted input. If the VSAM KSDS is reorganized with a sort sequence that changes the effective key ordering for records with duplicate high-order key fields (rare but possible with certain REORG parameter settings), the MATCH facility may generate incorrect unmatched output for records that should have matched. Work log: “CA Easytrieve Plus 11; PRMMATCH program; MATCH FILE BILLMST KEYS POLICY_GRP_NBR FILE ADJFILE KEYS POLICY_GRP_NBR; BILLMST VSAM KSDS REORG with REUSE(YES) parameter: 847 records with POLICY_GRP_NBR 'GRP-2024-*' now appear in different physical sequence within KSDS control intervals (REUSE(YES) does not respect FSPC free space distribution — records compacted, sequence within duplicates changes); MATCH generates 847 FALSE UNMATCHED output records for GRP-2024-* group numbers; premium adjustment report shows 847 group adjustments as unprocessed; diagnosis: IDCAMS PRINT CLUSTER(KEYSTONE.BILLING.MASTER) COUNT(1000) confirms resequencing; fix: add POLICY_GRP_NBR secondary sort key (SORT FILE BILLMST FIELDS POLICY_GRP_NBR ADJFILE FIELDS POLICY_GRP_NBR within PRMMATCH before MATCH); MATCH unmatched false positives: 847 → 0; hours: 3h diagnosis + 1h MATCH SORT clause addition.”

How HourTab tracks CA Easytrieve developer retainer hours

The FILE BILLMST VSAM KSDS UPDATE bug is invisible in batch-only testing environments because the conflict only manifests when CICS and the Easytrieve batch program run concurrently. In the original deployment of PRMVLD01 in 2019, the company ran CICS in a business-hours-only schedule: CICS started at 7:00 AM and shut down at 8:00 PM. PRMVLD01 ran from 2:00 AM to 4:37 AM, well within the CICS shutdown window. No conflict occurred. The UPDATE attribute in the FILE statement was a minor deviation from best practice but produced no observable problem. When the company transitioned to 24/7 CICS operations in 2024 to support an online self-service portal for employer benefits administrators, PRMVLD01 continued to run on its unchanged schedule — but now CICS was running at 2:00 AM. For 10 months, the BILLMST conflict happened every 28th of the month and was attributed to “CICS performance issues during month-end” until the Easytrieve developer was brought in specifically to investigate the pattern.

The work log entry that makes Easytrieve VSAM access work auditable must name every element: the Easytrieve version (CA Easytrieve Plus 11), the program name (PRMVLD01), the FILE statement attribute that caused the conflict (FILE BILLMST VSAM KSDS UPDATE), the VSAM cluster name (KEYSTONE.BILLING.MASTER), the SHAREOPTIONS setting (SHROPTNS(1-3) from IDCAMS LISTCAT), the CICS file name (BILLMST), the CICS error observed (EIBRESP=NOTOPEN, CICS file CLOSED/DISABLED), the root-cause finding (zero PUT BILLMST in source — UPDATE was speculative), the fix (FILE BILLMST VSAM KSDS INPUT in PRMVLD01 SYSIN), and the zero-error outcome (847/month NOTOPEN rate to 0). A log that says “fixed Easytrieve file access issue, 4h” is not auditable to the benefits administrator’s IT director who needs to explain to the employer group clients why their billing portal was showing errors every month-end for 10 months. HourTab gives Easytrieve developers a public retainer-hours URL they send to clients — health insurance organizations, banks, government agencies, and benefits administrators that run critical monthly Easytrieve batch programs against VSAM masters and Db2 tables maintained by an Easytrieve specialist on monthly retainer.

The broader context for Easytrieve retainer billing is that a FILE VSAM KSDS UPDATE specifying write access that a read-only program never uses is the Easytrieve instantiation of the same class of invisible access-mode over-specification that creates contention throughout the mainframe batch/online stack. The JCL developer retainer post covers the same pattern at the JCL layer: PREMBIL1 STEP020 DISP=OLD (exclusive access) on PREM.BILLING.MASTER.FILE when PREMBIL2 only performs COBOL OPEN INPUT — the fix is DISP=SHR, same principle. The IBM i developer retainer post covers the lock release pattern at the ILE RPG layer: CHAIN without UNLOCK holds 847 record locks on INVMST because the program never issued UPDATE or UNLOCK after acquiring the lock. In each case, the root cause is over-claiming: claiming more access (exclusive open, write-capable VSAM ACB, held record lock) than the program actually uses, and failing to release it in the expected timeframe. Easytrieve’s FILE statement INPUT vs. UPDATE is the Easytrieve-specific control point for this over-claiming pattern.

FAQ: CA Easytrieve developer retainers