VSAM developer on retainer: LOANUPD FILE STATUS 97 on OPEN I-O LOANMSTR after STOP RUN without CLOSE on validation failure — IDCAMS VERIFY resets VSAM close error — IBM VSAM developer, VSAM KSDS developer on monthly retainer

z/OS 2.5 • VSAM KSDS • AGRI.LOANMSTR.KSDS • Enterprise COBOL LOANUPD • FILE STATUS 97 • IDCAMS LISTCAT • IDCAMS VERIFY • retainer billing

At Great Plains Agrifinance, the nightly loan balance update job LOANAJB began failing with a self-perpetuating error. The Enterprise COBOL program LOANUPD opened the VSAM KSDS AGRI.LOANMSTR.KSDS for I-O, immediately received FILE STATUS = 97, displayed OPEN ERROR 97 - LOANMSTR to SYSOUT, and called STOP RUN — which terminated the job without closing the dataset. The next night, LOANAJB ran again, received FILE STATUS 97 again, and called STOP RUN again. The dataset stayed in a permanently broken state. Operations staff discovered it would not self-recover, escalated to the VSAM developer retainer, who submitted an IDCAMS VERIFY job to reset the catalog flag, and reran LOANAJB. Three months later, the same sequence repeated. Then again two months after that.

The VSAM developer retainer was being called in to run the same VERIFY recovery procedure over and over — because the root cause had never been addressed. LOANUPD had five STOP RUN calls in its processing loop, on error paths triggered by invalid loan types, matured loans, records not found, and REWRITE errors. Not one of them was preceded by CLOSE LOANMSTR. Each time any of those paths fired, VSAM marked AGRI.LOANMSTR.KSDS with a close-error flag in the catalog. The next LOANUPD run hit FILE STATUS 97, called STOP RUN without CLOSE, and re-triggered the same flag.

VSAM KSDS OPEN I-O and the close-error catalog flag

VSAM (Virtual Storage Access Method) is the primary structured file system for z/OS batch applications. A VSAM KSDS (Key-Sequenced Data Set) organises records in key order, with both a data component and an index component. Enterprise COBOL and COBOL II programs access VSAM KSDS datasets through the standard COBOL file I/O model: a SELECT statement in the ENVIRONMENT DIVISION designates the file as ORGANIZATION IS INDEXED, and READ, WRITE, REWRITE, and DELETE verbs perform record-level operations.

VSAM maintains a catalog entry for each cluster in the VSAM catalog (the ICF catalog). Part of that catalog entry is a set of flags that track the operational state of the dataset. One critical flag is the open indicator: when a program opens a VSAM dataset with OPEN I-O (or OPEN INPUT or OPEN OUTPUT), VSAM sets this indicator in the catalog entry to mark the dataset as currently open. When the program closes the dataset with CLOSE, VSAM clears the indicator. If the program terminates — via STOP RUN, a program check, an ABEND, or a JES2 job cancel — without executing CLOSE, z/OS closes the underlying file handles at the OS level, but the VSAM catalog’s open indicator is not automatically reset.

On the next attempt to open the dataset with OPEN I-O, VSAM detects the open indicator still set in the catalog, concludes that the dataset was not properly closed by its previous opener, and returns FILE STATUS = 97 to the COBOL program. FILE STATUS 97 means: “VSAM open failed — this dataset was not properly closed by its previous open. Run IDCAMS VERIFY to reset the catalog entry before attempting to open again.”

VSAM sets this flag as a data integrity precaution. A dataset that was not properly closed may have in-progress updates that were not committed to the data component, or an index that is out of sync with the data component. VSAM’s default posture is to refuse the OPEN until IDCAMS VERIFY has checked the structural integrity of the cluster and explicitly cleared the flag. This is correct behaviour — but it means that any COBOL program that calls STOP RUN without CLOSE will leave its VSAM datasets in a state that blocks all future OPEN I-O attempts until manual recovery.

LOANUPD and the five STOP RUN exits without CLOSE

LOANUPD was an Enterprise COBOL program that processed a daily loan balance update input file from the agrifinance lending system. For each record in the input, it read the corresponding loan record from AGRI.LOANMSTR.KSDS, validated the loan status, calculated the new balance, and REWRITE’d the updated record. The program structure:

ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
    SELECT LOANMSTR ASSIGN TO LOANMSTR
        ORGANIZATION IS INDEXED
        ACCESS MODE IS DYNAMIC
        RECORD KEY IS LM-LOAN-NUM
        FILE STATUS IS WS-FILE-STATUS.
...
PROCEDURE DIVISION.
1000-MAIN.
    OPEN I-O LOANMSTR
    IF WS-FILE-STATUS NOT = '00'
        DISPLAY 'OPEN ERROR ' WS-FILE-STATUS ' - LOANMSTR'
        STOP RUN                          *> EXIT A: if open failed, no CLOSE needed — safe
    END-IF

    PERFORM 2000-PROCESS-LOANS
        UNTIL WS-END-OF-INPUT

    CLOSE LOANMSTR
    STOP RUN.

2000-PROCESS-LOANS.
    READ LOAN-INPUT AT END
        SET WS-END-OF-INPUT TO TRUE
    NOT AT END
        PERFORM 3000-VALIDATE-AND-UPDATE
    END-READ.

3000-VALIDATE-AND-UPDATE.
    IF WS-LOAN-TYPE NOT = 'AG' AND WS-LOAN-TYPE NOT = 'EQ'
        DISPLAY 'INVALID LOAN TYPE: ' WS-LOAN-NUM
        STOP RUN                          *> EXIT B: BUG — CLOSE LOANMSTR missing

    IF WS-MATURITY-DATE < WS-TODAY
        DISPLAY 'MATURED LOAN SKIP: ' WS-LOAN-NUM
        PERFORM 9000-WRITE-ERROR-RECORD
        STOP RUN                          *> EXIT C: BUG — CLOSE LOANMSTR missing

    READ LOANMSTR KEY IS LM-LOAN-NUM
    EVALUATE WS-FILE-STATUS
        WHEN '23'
            DISPLAY 'LOAN NOT IN MASTER: ' WS-LOAN-NUM
            STOP RUN                      *> EXIT D: BUG — CLOSE LOANMSTR missing
        WHEN '00'
            CONTINUE
        WHEN OTHER
            DISPLAY 'READ ERROR: ' WS-FILE-STATUS ' LOAN ' WS-LOAN-NUM
            STOP RUN                      *> EXIT E: BUG — CLOSE LOANMSTR missing
    END-EVALUATE

    PERFORM 4000-CALC-NEW-BALANCE

    REWRITE LOANMSTR-RECORD
    IF WS-FILE-STATUS NOT = '00'
        DISPLAY 'REWRITE ERROR: ' WS-FILE-STATUS ' LOAN ' WS-LOAN-NUM
        STOP RUN                          *> EXIT F: BUG — CLOSE LOANMSTR missing
    END-IF.

Exits B through F all call STOP RUN inside the processing loop 2000-PROCESS-LOANS, which is called after OPEN I-O LOANMSTR in 1000-MAIN. At the moment any of those exits triggers, LOANMSTR is open. STOP RUN jumps past the CLOSE LOANMSTR statement in the normal exit path of 1000-MAIN. VSAM’s open indicator stays set. The catalog records AGRI.LOANMSTR.KSDS as improperly closed.

Exit A is the only safe one: if OPEN I-O returns FILE STATUS 97 (or any other non-00 status), the dataset was never successfully opened, so CLOSE is not needed and would produce a different error. The code as written handles exit A correctly. The problem is exits B through F, which fire inside the processing loop after the dataset has been opened and while it is in active use.

The self-perpetuating failure: FILE STATUS 97 blocks OPEN, STOP RUN re-triggers the flag

The first occurrence that triggered the cycle was a loan record with WS-LOAN-TYPE = 'CO' (commercial — not in LOANUPD’s scope for this particular batch file). Exit B fired, STOP RUN was called without CLOSE, and AGRI.LOANMSTR.KSDS was left with its open indicator set. The next night’s LOANAJB run reached OPEN I-O LOANMSTR and received FILE STATUS 97. The program’s exit A handler checked IF WS-FILE-STATUS NOT = '00' — true — displayed OPEN ERROR 97 - LOANMSTR, and called STOP RUN again. Exit A is the one safe STOP RUN, because at that point LOANMSTR was never successfully opened. But what exit A did not do was leave the VSAM catalog in any better state: the open indicator was already set from the previous night’s improper close, and exit A’s STOP RUN neither set nor cleared it. VSAM still saw the flag from the night before and would see it again the next night.

The cycle: night 1 (exit B fires, STOP RUN without CLOSE, open flag set) → VERIFY + rerun clears flag → night 2 runs successfully → weeks pass → night N (exit C fires on a matured loan, STOP RUN without CLOSE, open flag set again) → night N+1 gets FILE STATUS 97 → VERIFY again. Each time, the retainer developer was called in for a 30–45 minute VERIFY session. The underlying LOANUPD source code was never examined.

IDCAMS LISTCAT: confirming the open flag in the VSAM catalog

Before submitting VERIFY, the diagnostic step is IDCAMS LISTCAT to confirm that the open flag is set and that no structural damage occurred:

//LISTCAT  JOB (LOANACCT),'LISTCAT LOANMSTR'
//STEP1    EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  LISTCAT ENTRIES(AGRI.LOANMSTR.KSDS) ALL
/*

The relevant section of SYSPRINT output from a cluster with the close-error flag set:

CLUSTER ------- AGRI.LOANMSTR.KSDS
    IN-CAT --- UCAT.AGRI
    HISTORY
      DATASET-OWNER -------- (NULL)
      CREATION --------- 2024/003
      EXPIRATION ------- 0000/000
    PROTECTION-PSWD -- (NULL)
    ASSOCIATIONS
      DATA ------------ AGRI.LOANMSTR.KSDS.DATA
      INDEX ----------- AGRI.LOANMSTR.KSDS.INDEX

  DATA ------- AGRI.LOANMSTR.KSDS.DATA
    ...
    STATISTICS
      REC-TOTAL ------------------- 48293
      REC-DELETED ----------------- 0
      REC-INSERTED ---------------- 0
      REC-UPDATED ----------------- 12847     (last successful run)
    ...
    STATUS
      OPEN-CLOSE ----------- OPEN        <<< dataset not properly closed
      CLOSEERROR ----------- YES
    ...

  INDEX ------ AGRI.LOANMSTR.KSDS.INDEX
    ...
    STATUS
      OPEN-CLOSE ----------- OPEN        <<< index component also flagged
      CLOSEERROR ----------- YES

The OPEN-CLOSE = OPEN and CLOSEERROR = YES flags on both the DATA and INDEX components confirm the improper close. Both components must show OPEN-CLOSE = CLOSED and CLOSEERROR = NO before OPEN I-O will succeed without FILE STATUS 97. The LISTCAT output also shows that REC-UPDATED = 12847 from the last successful run — confirming that the dataset contents are intact from that run, and the current close-error state is a catalog flag issue only, not a data loss event.

IDCAMS VERIFY: resetting the catalog entry

IDCAMS VERIFY scans the VSAM cluster to confirm structural integrity (checking the index against the data component, verifying CI and CA boundaries, confirming that the high-used RBA values are consistent), then resets the open indicator and CLOSEERROR flag in the catalog:

//VERIFY   JOB (LOANACCT),'VERIFY LOANMSTR'
//STEP1    EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  VERIFY DATASET(AGRI.LOANMSTR.KSDS)
/*

Successful SYSPRINT output:

IDCAMS  SYSTEM SERVICES
...
IGG0CLEV  VSAM CATALOG VERIFY FOR AGRI.LOANMSTR.KSDS
  CLOSE ERROR INDICATOR RESET
  DATA COMPONENT: AGRI.LOANMSTR.KSDS.DATA — NO ERRORS FOUND
  INDEX COMPONENT: AGRI.LOANMSTR.KSDS.INDEX — NO ERRORS FOUND

IDC0001I FUNCTION COMPLETED, HIGHEST CONDITION CODE WAS 0

After VERIFY, a follow-up LISTCAT confirms OPEN-CLOSE = CLOSED and CLOSEERROR = NO on both components. LOANAJB can now be rerun; LOANUPD’s OPEN I-O returns FILE STATUS 00 and processing proceeds normally — until the next time an error-path STOP RUN fires without CLOSE.

The fix: CLOSE LOANMSTR before every STOP RUN on error exit paths

The permanent fix adds CLOSE LOANMSTR before each of the five error-path STOP RUN calls in 3000-VALIDATE-AND-UPDATE. The corrected paragraphs:

3000-VALIDATE-AND-UPDATE.
    IF WS-LOAN-TYPE NOT = 'AG' AND WS-LOAN-TYPE NOT = 'EQ'
        DISPLAY 'INVALID LOAN TYPE: ' WS-LOAN-NUM
        CLOSE LOANMSTR                    *> Added
        STOP RUN

    IF WS-MATURITY-DATE < WS-TODAY
        DISPLAY 'MATURED LOAN SKIP: ' WS-LOAN-NUM
        PERFORM 9000-WRITE-ERROR-RECORD
        CLOSE LOANMSTR                    *> Added
        STOP RUN

    READ LOANMSTR KEY IS LM-LOAN-NUM
    EVALUATE WS-FILE-STATUS
        WHEN '23'
            DISPLAY 'LOAN NOT IN MASTER: ' WS-LOAN-NUM
            CLOSE LOANMSTR                *> Added
            STOP RUN
        WHEN '00'
            CONTINUE
        WHEN OTHER
            DISPLAY 'READ ERROR: ' WS-FILE-STATUS ' LOAN ' WS-LOAN-NUM
            CLOSE LOANMSTR                *> Added
            STOP RUN
    END-EVALUATE

    PERFORM 4000-CALC-NEW-BALANCE

    REWRITE LOANMSTR-RECORD
    IF WS-FILE-STATUS NOT = '00'
        DISPLAY 'REWRITE ERROR: ' WS-FILE-STATUS ' LOAN ' WS-LOAN-NUM
        CLOSE LOANMSTR                    *> Added
        STOP RUN
    END-IF.

Exit A in 1000-MAIN is left unchanged: if OPEN I-O fails, LOANMSTR was never successfully opened and CLOSE must not be called. All other exits now close LOANMSTR before stopping. VSAM will set the open indicator at OPEN time and clear it at CLOSE time — so even if LOANUPD terminates via one of the error paths, the catalog records the dataset as properly closed.

Additionally, a defensive DECLARATIVES section was added to catch any unexpected VSAM I/O exception not covered by the explicit FILE STATUS checks:

PROCEDURE DIVISION.
DECLARATIVES.
LOANMSTR-ERROR SECTION.
    USE AFTER EXCEPTION CONDITION LOANMSTR.
    DISPLAY 'UNHANDLED LOANMSTR I/O ERROR: ' WS-FILE-STATUS
    CLOSE LOANMSTR
    STOP RUN.
END DECLARATIVES.

The DECLARATIVES handler fires on any VSAM I/O exception on LOANMSTR that is not handled by an explicit AT END or INVALID KEY clause. It closes the dataset before stopping, ensuring the catalog is left in a clean state regardless of the error type.

After compiling and deploying the corrected LOANUPD, five test scenarios were run against a test instance of AGRI.LOANMSTR.KSDS: one normal completion, one invalid loan type (exit B), one matured loan (exit C), one loan not in master (exit D), and one simulated REWRITE error (exit F). After each scenario, IDCAMS LISTCAT confirmed OPEN-CLOSE = CLOSED and CLOSEERROR = NO. No VERIFY was needed after any run. Over the four months following deployment: zero FILE STATUS 97 incidents, zero IDCAMS VERIFY emergency sessions, zero LOANAJB overnight failures from VSAM close errors.

How HourTab helps VSAM developer retainers

The Great Plains Agrifinance engagement spanned two phases. Recurring VERIFY sessions (before root cause fix): approximately 12 sessions at 0.5–0.75 hours each — IDCAMS LISTCAT confirming the open flag, VERIFY submission, FILE STATUS 00 confirmation, LOANAJB manual rerun. Root cause fix (one session): 0.5h LOANAJB job log analysis and FILE STATUS 97 identification, 0.5h IDCAMS LISTCAT diagnosis and VERIFY recovery, 1.5h LOANUPD source review (tracing all six STOP RUN exits, identifying which require CLOSE), 1.0h Enterprise COBOL fix (five CLOSE additions plus DECLARATIVES section), 1.0h regression testing (five test cases with LISTCAT after each), 0.5h documentation = 5.5h.

In a HourTab retainer log, the fix session reads: “LOANUPD FILE STATUS 97 root cause fix: IDCAMS LISTCAT confirmed CLOSEERROR=YES on both DATA and INDEX components of AGRI.LOANMSTR.KSDS; FILE STATUS 97 source traced to 5 STOP RUN exits in 3000-VALIDATE-AND-UPDATE without preceding CLOSE LOANMSTR (exits B–F: invalid loan type, matured loan, loan not in master, READ error, REWRITE error); fix: CLOSE LOANMSTR added before each of the 5 error STOP RUN calls; DECLARATIVES section added for unhandled I/O exceptions; compiled and tested 5 scenarios; LISTCAT after each: OPEN-CLOSE=CLOSED, CLOSEERROR=NO; zero FILE STATUS 97 incidents and zero VERIFY sessions in the 4 months since deployment; 5.5h.” Great Plains Agrifinance’s IT manager can compare the 5.5-hour fix session against the 6–9 hours of recurring VERIFY sessions it replaced, and immediately see the value of the retainer’s root-cause work.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: VSAM developer retainers

What does a VSAM developer on retainer typically do?

A VSAM developer on monthly retainer covers VSAM KSDS, ESDS, and RRDS application maintenance on z/OS, including Enterprise COBOL and COBOL II programs that access VSAM through SELECT/ASSIGN and READ/WRITE/REWRITE/DELETE verbs. Typical retainer work includes: LOANUPD-class COBOL batch program maintenance — diagnosing and fixing STOP RUN without CLOSE on error exit paths (FILE STATUS 97 on next OPEN I-O), COBOL FILE STATUS analysis (97 for close error, 22 for duplicate key, 23 for record not found, 35 for dataset not found), and IDCAMS VERIFY for close-error recovery; VSAM cluster administration via IDCAMS (DEFINE CLUSTER, ALTER, REPRO, DELETE, LISTCAT, PRINT); JCL DD statement maintenance including AMP parameter tuning (BUFSP, BUFND, BUFNI); DFSMS management class and storage class assignment review; and VSAM SHAREOPTIONS analysis for clusters accessed from both batch jobs and CICS online programs.

What VSAM work is most commonly underlogged on retainer?

STOP RUN without CLOSE on error exit paths — producing FILE STATUS 97 on subsequent OPEN I-O — is the most systematically underlogged VSAM retainer work. Pattern: COBOL batch program opens VSAM KSDS for I-O; a validation failure triggers STOP RUN without preceding CLOSE; VSAM marks the cluster with CLOSEERROR=YES in the catalog; the next run gets FILE STATUS 97 and also calls STOP RUN without CLOSE; self-perpetuating failure cycle requires IDCAMS VERIFY to break. Each VERIFY session is a recurring retainer ticket. The fix: CLOSE the VSAM dataset before every STOP RUN and GOBACK on all error exit paths. IDCAMS LISTCAT ALL is the diagnostic (OPEN-CLOSE=OPEN and CLOSEERROR=YES on DATA and INDEX components confirms the condition). IDCAMS VERIFY resets the flags and allows normal OPEN.

What are typical VSAM developer retainer rates?

Entry-level VSAM developers with COBOL batch basics (SELECT ASSIGN ORGANIZATION IS INDEXED, READ/WRITE/REWRITE/DELETE, FILE STATUS '00'/'22'/'23'/'35' checking; basic IDCAMS DEFINE CLUSTER and REPRO) typically bill at $70 to $120 per hour. Mid-level VSAM developers with close-error diagnosis and recovery (FILE STATUS 97, IDCAMS LISTCAT CLOSEERROR flag, IDCAMS VERIFY, CLOSE-before-STOP-RUN fix), VSAM cluster administration (IDCAMS ALTER, DELETE/DEFINE for reorganization, REPRO for migration), JCL AMP tuning (BUFSP, BUFND, BUFNI), DFSMS storage class review, and VSAM SHAREOPTIONS analysis for batch-CICS sharing typically bill at $100 to $170 per hour. Senior VSAM developers with DFSMS architecture (ACS routines, storage class definitions, management class tiering), cross-system VSAM sharing design (GRS ENQ scope, SHAREOPTIONS 4 3 for sysplex), VSAM performance engineering (CI/CA size optimization, free space tuning), and reorganization planning typically bill at $150 to $260 per hour. Monthly retainer ranges: $4,000 to $9,000 per month for COBOL batch VSAM maintenance and close-error diagnosis; $7,000 to $14,000 per month for VSAM cluster architecture and DFSMS management.

What should a VSAM developer retainer agreement include?

A VSAM retainer agreement should specify: z/OS and DFSMS version; which VSAM clusters are in scope by dataset name and type (KSDS, ESDS, RRDS); whether COBOL batch program maintenance is in scope and which programs use VSAM; whether IDCAMS VERIFY recovery sessions count separately from programming tasks (VERIFY sessions during production incidents typically run 0.5–1h); whether VSAM cluster administration is included (DEFINE CLUSTER, ALTER, REPRO, DELETE, LISTCAT); whether reorganization planning and execution is in scope (REPRO to alternate cluster, DELETE and DEFINE, REPRO back); whether DFSMS storage class and management class changes are covered; whether VSAM SHAREOPTIONS analysis for batch-CICS sharing is included — SHAREOPTIONS has significant data integrity implications; and whether VSAM performance tuning (CI/CA size, buffer allocation, AMP parameter optimization) is within scope.

How should VSAM developer retainer hours be logged?

Log each VSAM retainer session with: z/OS version, DFSMS version, cluster name (AGRI.LOANMSTR.KSDS), cluster type (KSDS), program name (LOANUPD), FILE STATUS value received (97), IDCAMS LISTCAT output (OPEN-CLOSE and CLOSEERROR flags on DATA and INDEX components), IDCAMS VERIFY submission and return code (0 for success), FILE STATUS on first OPEN I-O after VERIFY (00), and COBOL fix applied. For FILE STATUS 97: z/OS 2.5; AGRI.LOANMSTR.KSDS (KSDS, 48,293 records); Enterprise COBOL LOANUPD in LOANLIB.LOADLIB; LOANAJB job; five STOP RUN exits in 3000-VALIDATE-AND-UPDATE without CLOSE (invalid loan type, matured loan, loan not in master, READ error, REWRITE error); LISTCAT: OPEN-CLOSE=OPEN, CLOSEERROR=YES on DATA and INDEX; VERIFY: CLOSE ERROR INDICATOR RESET, no structural errors; post-VERIFY OPEN I-O: FILE STATUS 00; fix: CLOSE LOANMSTR added before 5 error exits + DECLARATIVES section for unhandled I/O; 5 test cases — all LISTCAT post-run: OPEN-CLOSE=CLOSED, CLOSEERROR=NO; hours: 0.5h log analysis + 0.5h LISTCAT diagnosis + 0.5h VERIFY recovery + 1.5h LOANUPD source review + 1.0h COBOL fix + 1.0h test + 0.5h documentation = 5.5h.

See also: COBOL developer retainer · Easytrieve developer retainer · Db2 developer retainer