DataStage developer on retainer: DS_Claims_Daily_Load Sequential File stage writes 8 partition files instead of 1 — downstream consumer glob fails to find daily_claims_YYYYMMDD.txt — IBM DataStage developer on monthly retainer

IBM InfoSphere DataStage 11.7.1.1 • Parallel Job • 8-node grid • Sequential File • Funnel Collect • partition suffix • GLCC_ETL • retainer billing

At Great Lakes Claims Clearinghouse, the nightly claims batch had been running without incident for fourteen months when it silently stopped delivering output to the trading partner interface layer. The DataStage job completed successfully every night — status green, row count correct, RC=0. But the downstream Python consumer script that picked up the daily claims file and submitted it to the clearinghouse network was logging the same error every morning at 03:47: FileNotFoundError: No files found matching pattern 'daily_claims_*.txt'. Four days of claims — 248,371 transactions — had not been delivered. The IBM DataStage developer retainer was called in on October 15. The DataStage job had not been modified. What had changed was the infrastructure: the DataStage grid had been expanded from a 1-node development-equivalent configuration to an 8-node production grid as part of a server refresh two weeks earlier. That expansion had silently changed the output of every Sequential File stage in every Parallel Job from one file to eight files.

Great Lakes Claims Clearinghouse ran IBM InfoSphere DataStage 11.7.1.1 on RHEL 8.3. The DataStage engine tier had been running on a single processing node (ds-engine-01) since the system was deployed in late 2024. In late September 2026, the infrastructure team provisioned a production-grade 8-node DataStage grid (ds-engine-01 through ds-engine-08) to handle growing claims volume. The existing jobs were migrated without modification to the new grid configuration. The GLCC_ETL project contained 23 DataStage Parallel Jobs processing daily claims, eligibility, and remittance files. The DS_Claims_Daily_Load job was the highest-volume job: it read approximately 62,000 claims per day from an Oracle 19c source, applied member validation lookups, and wrote the enriched output to a single flat file in /data/outbound/claims/ for pickup by the Python consumer.

How DataStage Parallel Jobs write Sequential File output on multi-node grids

IBM DataStage Parallel Jobs process data in parallel across all configured processing nodes. Each stage in a Parallel Job runs simultaneously on all N nodes, processing its partition of the total data volume. A job on an 8-node grid runs 8 instances of each stage, each handling approximately 1/8 of the total rows (the exact split depends on the partition method used for each stage).

The Sequential File stage is the standard DataStage output stage for writing flat files. In a Parallel Job running on multiple nodes, if the Sequential File stage has no Funnel stage before it to collect all partitions onto a single node, each node writes its own partition of the data to a separate file. The filename receives a node-index suffix automatically appended by the DataStage engine: .0 for node 0, .1 for node 1, through .7 for node 7 on an 8-node grid. The result is 8 output files instead of 1:

-- Output directory after migration to 8-node grid:
$ ls -lh /data/outbound/claims/daily_claims_20261015.txt*

-rw-r--r-- 1 dsadm dstage  982K  Oct 15 02:41  daily_claims_20261015.txt.0   (7,891 rows)
-rw-r--r-- 1 dsadm dstage  967K  Oct 15 02:41  daily_claims_20261015.txt.1   (7,771 rows)
-rw-r--r-- 1 dsadm dstage  994K  Oct 15 02:41  daily_claims_20261015.txt.2   (7,993 rows)
-rw-r--r-- 1 dsadm dstage  978K  Oct 15 02:41  daily_claims_20261015.txt.3   (7,861 rows)
-rw-r--r-- 1 dsadm dstage  986K  Oct 15 02:41  daily_claims_20261015.txt.4   (7,928 rows)
-rw-r--r-- 1 dsadm dstage  969K  Oct 15 02:41  daily_claims_20261015.txt.5   (7,791 rows)
-rw-r--r-- 1 dsadm dstage  991K  Oct 15 02:41  daily_claims_20261015.txt.6   (7,924 rows)
-rw-r--r-- 1 dsadm dstage  973K  Oct 15 02:41  daily_claims_20261015.txt.7   (7,808 rows)

Total: 8 files, 7,844,592 bytes, 62,967 rows combined

-- DataStage Director job log:
DS_Claims_Daily_Load   2026-10-15 02:38:11 → 02:41:44   Status: Finished   Rows: 62,967   RC: 0

The DataStage job completed successfully and wrote all 62,967 rows. But the Python consumer script used a glob pattern to find the file:

-- Python consumer: claims_submit.py (downstream consumer)
import glob
import os

outbound_dir = '/data/outbound/claims/'
pattern = 'daily_claims_*.txt'           # ← matches *.txt only, not *.txt.0 through *.txt.7

files = glob.glob(os.path.join(outbound_dir, pattern))
if not files:
    raise FileNotFoundError(f"No files found matching pattern '{pattern}' in {outbound_dir}")

# ... submit files to clearinghouse network

The glob pattern daily_claims_*.txt matched files ending in .txt. The 8 files created by DataStage ended in .txt.0 through .txt.7 — none matched the pattern. The Python consumer had been running successfully for 14 months on the 1-node grid, where DataStage wrote a single file named daily_claims_YYYYMMDD.txt with no suffix. The transition to the 8-node grid introduced the partition suffixes with no change to any job definition or downstream script.

Diagnosis: DataStage Director log and output directory inspection

The DataStage developer retainer connected to the DataStage Designer and reviewed the DS_Claims_Daily_Load job design. The job flow was straightforward:

-- DS_Claims_Daily_Load Parallel Job stage flow (before fix):
[Oracle Connector]           -- reads CLAIMS table from Oracle 19c (~62K rows/day)
        |
[Transformer: TRF_Validate]  -- 23 derivation columns: member validation, date normalization
        |
[Lookup: LKP_Provider]       -- lookup against PROVIDER_MASTER dataset (1.2M rows)
        |
[Transformer: TRF_Enrich]    -- 8 output columns mapped to output schema
        |
[Sequential File: SQ_Output] -- writes to /data/outbound/claims/daily_claims_#date#.txt
                                 (NO Funnel stage before this stage)

The Sequential File stage SQ_Output had its File property set to /data/outbound/claims/daily_claims_#date#.txt using a DataStage job parameter #date# that resolved to the current date in YYYYMMDD format. There was no Funnel stage between the final Transformer and the Sequential File stage. On a 1-node grid, this design produced a single file. On an 8-node grid, it produced 8 files.

The DataStage Director log for all four affected days showed identical patterns: Status: Finished, RC: 0. The output directory confirmed 8 files per day (32 files total for the 4-day period), all correctly named with the date parameter and partition suffix. The Python consumer log showed FileNotFoundError at 03:47 AM each day.

A listing of the output directory also revealed that several other GLCC_ETL jobs had the same issue: the remittance file job and the eligibility file job both used Sequential File output stages without Funnel stages, and both had been producing 8 files since the grid migration. Those consumers had not yet triggered failures only because they polled on a weekly rather than daily schedule.

The fix: adding a Funnel stage in Collect mode before the Sequential File stage

In IBM DataStage, a Funnel stage is a processing stage that combines multiple partitioned data streams into a single ordered or unordered stream. The Funnel stage has three operating modes:

• Continuous — outputs rows from each partition as they arrive, in undefined order. Fastest for downstream stages that do not require a specific row sequence.
• Sort — merges sorted partitions using a merge-sort algorithm, producing a globally sorted output stream.
• Collect — outputs all rows from partition 0 first, then all rows from partition 1, and so on through partition N-1. Preserves intra-partition order. Produces a single sequential data stream on node 0.

For the DS_Claims_Daily_Load fix, a Funnel stage in Collect mode was added between the final Transformer stage and the Sequential File stage. Collect mode assembles all 8 node partitions into a single stream on node 0, guaranteeing that the Sequential File stage sees exactly 1 partition and writes exactly 1 file:

-- DS_Claims_Daily_Load Parallel Job stage flow (after fix):
[Oracle Connector]
        |
[Transformer: TRF_Validate]
        |
[Lookup: LKP_Provider]
        |
[Transformer: TRF_Enrich]
        |
[Funnel: FNL_Collect]        -- NEW: mode=Collect; all 8 partitions → 1 stream on node 0
        |
[Sequential File: SQ_Output] -- now sees 1 partition → writes 1 file: daily_claims_YYYYMMDD.txt
                                 (no partition suffix appended)

The Funnel stage properties in DataStage Designer:

-- DataStage Designer: Funnel stage FNL_Collect properties
Stage type:   Funnel
Mode:         Collect
Sort keys:    (none — CLAIM_DATE DESC was considered but not required for downstream format)
Partitioning: Auto (inherits partitioning from upstream stage)

-- Note: Collect mode does NOT require pre-sorted input.
-- Rows from each partition are collected in partition-index order (0 first, then 1, ..., 7).
-- The output is deterministic: same data, same partition assignments → same output order.
-- For the clearinghouse submission format, row order within the file is not specified,
-- so Collect (vs. Sort) is correct and faster.

The job was recompiled and run in the DataStage QA environment. The test run output:

-- QA test run output directory:
$ ls -lh /data/outbound/claims/daily_claims_20261016.txt*

-rw-r--r-- 1 dsadm dstage  7.7M  Oct 16 11:22  daily_claims_20261016.txt   (61,293 rows)

-- DataStage Director QA run:
DS_Claims_Daily_Load   2026-10-16 11:19:07 → 11:22:41   Status: Finished   Rows: 61,293   RC: 0

-- Python consumer test run:
files = glob.glob('/data/outbound/claims/daily_claims_*.txt')
# Returns: ['/data/outbound/claims/daily_claims_20261016.txt']
# Submitting 61,293 claims to clearinghouse...
# Submission result: accepted, 61,293 rows confirmed.

A single file, no partition suffix, matched by the Python consumer glob. Row count: 61,293 (the test day’s volume). Python consumer submitted successfully. The job was deployed to production.

The remittance and eligibility file jobs were also audited. Both had Sequential File output stages without Funnel stages. The same Funnel Collect fix was applied to both jobs before either consumer encountered a failure. The four-day backlog of claims (248,371 rows across 4 days, in 32 partition files) was consolidated manually by the DataStage developer into 4 single files and submitted to the clearinghouse network by the operations team. All 248,371 claims were accepted by the clearinghouse; no claim was lost or duplicated.

Why partition output bugs are a recurring DataStage retainer problem

IBM DataStage Parallel Job design is fundamentally parallel: every stage is designed to run on all N nodes simultaneously. The default behavior of the Sequential File stage in a Parallel Job environment is multi-file output, because single-file output requires either a Funnel stage (which adds a serialization step) or explicit configuration. A job designed and tested on a 1-node development or QA grid behaves identically on production because N=1 means 1 file with no partition suffix. The multi-node parallel output behavior only appears when the job is deployed to a multi-node grid — which is exactly the transition that triggers the issue most often.

The DataStage job log provides no warning for this pattern. The Sequential File stage writes its assigned partition of rows to the expected filename plus the partition suffix, and reports the correct row count. RC=0. The partition suffix behavior is documented but not flagged as a potential configuration issue in the job log. The only detection method is inspecting the output directory directly and verifying that the file count matches what downstream consumers expect.

A DataStage developer retainer who performs a grid migration review — specifically auditing all Sequential File output stages in all Parallel Jobs for Funnel stage presence before approving the migration — catches this before any downstream consumer fails. For existing deployments, the audit is: grep all job designs for Sequential File stages not preceded by a Funnel stage, then verify that the downstream consumer for each such job can handle N files or has a glob pattern that accommodates the partition suffix. The audit takes 2–4 hours per DataStage project and is a standard retainer deliverable during any grid infrastructure change.

How HourTab helps DataStage developer retainers

The Great Lakes Claims Clearinghouse engagement totaled 4.5 hours: diagnosis (1.5h — 0.5h reviewing the DataStage Director log for all four failed days to confirm RC=0 and correct row counts, listing the output directory to find the 32 partition files, and confirming the Python consumer error message against the partition-suffixed filenames; 0.5h tracing the grid migration date in the infrastructure change log and correlating the 8-node grid activation on September 28 with the first Python consumer failure on October 12; 0.5h reviewing the DS_Claims_Daily_Load job design in DataStage Designer to confirm no Funnel stage precedes the Sequential File output stage, and auditing the remittance and eligibility file jobs for the same pattern) and fix (3.0h — 1.0h adding Funnel stage in Collect mode to DS_Claims_Daily_Load, recompiling and running QA test: single-file output confirmed, Python consumer test run confirmed, row count 61,293 matches Oracle source; 0.5h applying the same Funnel Collect fix to the remittance and eligibility file jobs (pre-emptive, no consumer failure yet); 0.5h production deployment of all three fixed jobs; 0.5h manual backlog consolidation: 32 partition files across 4 days merged into 4 single-day files by the developer using a shell cat command and submitted to the clearinghouse via the Python consumer; 0.5h documentation: post-incident write-up for retainer file, sequential file audit SOP added to grid migration checklist, and retainer review note added for all remaining GLCC_ETL jobs with Sequential File output stages).

In a HourTab retainer log, the incident session reads: “DS_Claims_Daily_Load consumer failure 4 days (Oct 12–15); DataStage Director: RC=0, 62,847–63,104 rows/day; output directory: 8 partition files daily_claims_YYYYMMDD.txt.0 through .7 per day (32 files total, 248,371 rows); Python consumer glob 'daily_claims_*.txt' matches *.txt only, not *.txt.0; root cause: 8-node grid migration Sep 28 — Sequential File output stage without Funnel stage → 8 files per run; fix: Funnel stage (mode=Collect) added before SQ_Output in DS_Claims_Daily_Load; same fix applied pre-emptively to remittance and eligibility file jobs; all 3 jobs recompiled and deployed; QA test: single file, correct row count, Python consumer success; backlog: 4 daily files consolidated from 32 partition files and submitted — all 248,371 claims accepted by clearinghouse; audit: 5 other GLCC_ETL jobs with Sequential File output stages — all now have Funnel Collect stages; 4.5h.” The operations director at Great Lakes Claims Clearinghouse sees the exact cause (grid migration + missing Funnel stage), the scope of data affected (248,371 claims, 4 days), the fix applied to all affected jobs (not just the one that failed), and the audit scope — with no ambiguity about how the retainer time was spent.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: IBM DataStage developer retainers

See also: Informatica developer retainer · Oracle DBA retainer · Cognos developer retainer