MuleSoft developer on retainer: lfsg-product-sync syncProductCatalogFlow Database connector SELECT without streaming loads 847,293-row PRODUCT_CATALOG into heap — java.lang.OutOfMemoryError crashes nightly CRM sync — 21 days of missed product catalog updates

Mule Runtime 4.4.0 EE • CloudHub 1.0 • Database connector • streaming=false • CursorIteratorProvider • fetchSize • OutOfMemoryError • retainer billing

At Lakewood Financial Services Group, the nightly product catalog sync had been running without incident for nearly eight months when it silently stopped updating the CRM. The Mule application completed each scheduled run — or rather, it tried to. But at 02:03 every morning, the CloudHub worker was quietly running out of heap memory in the middle of the Database SELECT that kicked off the sync flow. The Mule runtime recovered from the java.lang.OutOfMemoryError each time — flows can throw OOM exceptions without crashing the runtime process — but syncProductCatalogFlow errored out with no retry, no alert, and no notification to anyone. Twenty-one consecutive nightly runs failed. The CRM system’s last successful product update was September 2. When a CRM analyst noticed on September 24 that 12,847 products from the Meadowbrook Trust acquisition were not visible in the quoting tool, the MuleSoft developer retainer was engaged the same day.

Lakewood Financial Services Group ran Mule Runtime 4.4.0 Enterprise Edition on CloudHub 1.0. The lfsg-product-sync application (version 1.2.0) had been deployed on January 14, 2026 on a 2 vCore CloudHub worker with a 3.5GB JVM heap maximum (-Xmx3584m). Its primary flow, syncProductCatalogFlow, ran on a daily cron schedule (0 0 2 * * ?, 02:00 each morning). The flow read every product record from the PRODUCT_CATALOG table in the LFSG_Operations SQL Server 2019 database, then iterated through the result with a ForEach scope that sent each product record to the CRM REST API via HTTP Request connector (https://crm.lfsg.internal/api/products/upsert). At deployment in January 2026, PRODUCT_CATALOG held 293,847 rows. The JVM heap was more than adequate.

Between January and August 2026, three acquisitions changed the landscape. Meadowbrook Trust, Pacific Regional Credit Union, and Summit Wealth Management were all absorbed into LFSG, and each institution’s product catalog was loaded into PRODUCT_CATALOG via nightly ETL from the source institution’s systems. By September 3, 2026, PRODUCT_CATALOG held 847,293 rows — a 188% increase from the deployment baseline. The syncProductCatalogFlow Database connector had no streaming attribute set. In Mule 4, that means streaming="false" is the default: the connector executes the SQL query and loads the entire result set into a List<Map<String, Object>> in the JVM heap before returning the payload to the flow. On the night of September 3, materializing 847,293 rows into that List required an estimated 2.29GB of heap. The JVM heap maximum was 3,584MB. With JVM overhead and ForEach iterator state, the heap was exhausted before the List construction finished.

The failure was invisible for 21 days because no error handler was configured on syncProductCatalogFlow, and no Runtime Manager alert had been set up for flow errors on the lfsg-product-sync application. The Mule runtime logged the java.lang.OutOfMemoryError to CloudHub application logs — but nobody was watching those logs at 02:03 AM. The first human signal was the CRM analyst’s report on September 24 that products were missing from the quoting tool. The retainer engagement began that morning.

How Mule 4 Database connector streaming works — and what happens when it is disabled

In Mule 4, the Database connector’s streaming attribute controls how the JDBC result set is returned to the flow. The behavior of the two modes is fundamentally different in terms of heap usage:

With streaming="false" (the default), the Database connector executes the SQL query and immediately iterates the entire JDBC ResultSet, copying every row into a List<Map<String, Object>> in the JVM heap. The JDBC connection is released after this materialization completes. The flow receives a fully-realized Java List as its payload. For a query returning 847,293 rows with 17 columns each (mix of VARCHAR, DECIMAL, DATE), each row materializes as a Map<String, Object> with estimated average size of 2.78KB including Java object overhead. Total heap allocation for the List: approximately 2.29GB.

With streaming="true", the Database connector returns a CursorIteratorProvider — a lazy wrapper around the open JDBC ResultSet. No rows are loaded into heap at payload creation time. The ForEach scope that iterates the payload opens a cursor from the CursorIteratorProvider and advances it one batch at a time, controlled by the fetchSize attribute which maps directly to Statement.setFetchSize() on the JDBC PreparedStatement. With fetchSize="1000", the SQL Server JDBC driver fetches 1,000 rows per network round-trip from the database server. At any point during iteration, only the current batch of rows occupies heap — not the entire result set.

<db:select doc:name="DB_Select_Products" config-ref="LFSG_DB_Config">
    <db:sql>SELECT * FROM PRODUCT_CATALOG</db:sql>
</db:select>






<db:select doc:name="DB_Select_Products" config-ref="LFSG_DB_Config" streaming="true" fetchSize="1000">
    <db:sql>SELECT * FROM PRODUCT_CATALOG ORDER BY PRODUCT_ID</db:sql>
</db:select>

The ORDER BY PRODUCT_ID clause was added to the streaming query for two reasons: first, SQL Server cursor-based fetches on an unordered query can return inconsistent row sequences across batches if concurrent writes occur; second, the CRM upsert API is idempotent but logs a warning for duplicate product IDs, so a deterministic processing order simplifies any partial-failure replay. For a read-only nightly sync with no concurrent writes, the ORDER BY is a minor overhead (the PRODUCT_ID column is the clustered index key on PRODUCT_CATALOG, so no additional sort operation occurs).

One important detail for SQL Server specifically: the Microsoft JDBC Driver for SQL Server has a default fetchSize of 0, which means “fetch all rows in a single server-side cursor batch.” On most JDBC drivers, a default fetchSize of 0 or 1 pairs acceptably with streaming because the driver still returns rows one or a few at a time. On the MSSQL JDBC driver, fetchSize=0 with a streaming cursor causes the driver to pre-fetch the entire result set into a server-side cursor cache before returning the first row to the client — which defeats the purpose of streaming for very large result sets. Setting fetchSize="1000" explicitly overrides this and ensures rows are returned in manageable 1,000-row batches.

Diagnosis: CloudHub logs, JVM heap dashboard, and SQL Server row count

The MuleSoft developer retainer began diagnosis in CloudHub Runtime Manager at 09:14 on September 24. The first stop was the application log for lfsg-product-sync.

-- CloudHub Runtime Manager → Applications → lfsg-product-sync → Logs
-- Filter: level=ERROR, date range Sep 3–Sep 23 2026

[2026-09-03 02:03:47.312] ERROR MuleRuntime/uber.08: ...syncProductCatalogFlow:
  java.lang.OutOfMemoryError: Java heap space
    at java.util.ArrayList.add(ArrayList.java:459)
    at org.mule.module.db.internal.result.resultset.ResultSetIterator.next(ResultSetIterator.java:71)
    at org.mule.module.db.internal.result.resultset.ListResultSetCloser.close(ListResultSetCloser.java:45)
    at org.mule.module.db.internal.result.resultset.SimpleResultSetHandler.processResultSet(...)
    at org.mule.module.db.internal.domain.executor.SelectExecutor.execute(SelectExecutor.java:52)
    ...

[2026-09-04 02:03:47.198] ERROR MuleRuntime/uber.08: ...syncProductCatalogFlow:
  java.lang.OutOfMemoryError: Java heap space
    at java.util.ArrayList.add(ArrayList.java:459)
    ...

-- Same stack trace at 02:03:47 (±3 seconds) every night Sep 3 through Sep 23.
-- 21 consecutive failures. No other errors in the log outside this window.
-- Application remained running (Mule runtime did not crash — OOM was caught at flow scope).

The stack trace pinpoints the failure: ArrayList.add called from ResultSetIterator.next inside ListResultSetCloser.close. The ListResultSetCloser is the Mule 4 Database connector’s internal class responsible for materializing the JDBC ResultSet into a List when streaming="false". The OOM occurs during List construction — confirming that no rows have been delivered to the flow yet when the failure occurs. The CRM sync has been producing zero updates since September 3.

The second diagnostic step was the JVM heap dashboard in Runtime Manager:

-- CloudHub Runtime Manager → Applications → lfsg-product-sync → Dashboard → JVM Heap Used
-- Time range: Sep 3 2026, 01:55 – 02:15

01:55  Heap used:  312 MB  (baseline: application idle, nightly GC completed)
02:00  Heap used:  318 MB  (syncProductCatalogFlow scheduled trigger fires)
02:01  Heap used:  692 MB  (Database SELECT executing; ArrayList.add in progress)
02:02  Heap used: 1,447 MB  (List construction ~50% complete)
02:03  Heap used: 2,891 MB  (List construction ~80% complete)
02:03:47  Heap used: 3,582 MB  (100.0% of -Xmx3584m — OOM thrown)
02:04  Heap used:  401 MB  (GC sweep after OOM catch; List partially constructed, abandoned)

-- Pattern repeats identically each night Sep 3 through Sep 23.
-- Heap climb: 312MB baseline → 3,582MB peak in 3m47s.
-- Post-OOM drop: GC reclaims abandoned ArrayList allocation immediately.

The heap dashboard confirmed the materialization pattern: steady climb from baseline to 100% heap in approximately 3 minutes and 47 seconds each night, followed by an immediate GC-driven drop after the OOM exception was thrown and the ArrayList reference was abandoned. The shape of the curve is characteristic of a single large allocation growing linearly — consistent with ArrayList.add in a loop over a JDBC result set.

The third diagnostic step was a direct row count query on the SQL Server database:

-- SQL Server 2019, LFSG_Operations database
-- Executed Sep 24 2026 09:31 AM by retainer developer

SELECT COUNT(*) AS row_count FROM PRODUCT_CATALOG;
-- Result: 847,293

-- Confirm deployment baseline (from Jan 14 2026 data migration report):
-- PRODUCT_CATALOG row count at go-live: 293,847

-- Growth breakdown:
-- Jan 14, 2026: 293,847  (LFSG original product catalog at deployment)
-- Mar 2026: +147,203     (Meadowbrook Trust product catalog ETL complete)
-- Jun 2026: +201,811     (Pacific Regional Credit Union product catalog ETL complete)
-- Aug 2026: +204,432     (Summit Wealth Management product catalog ETL complete)
-- Sep 3, 2026: 847,293   (first night OOM threshold crossed)

-- Memory estimate:
-- 847,293 rows × 17 columns × avg column size + Java Map/ArrayList object overhead
-- = 847,293 × ~2.78KB per Map entry
-- = ~2.29GB for List> allocation
-- Plus JVM baseline (312MB) + ForEach iterator state + HTTP connector thread pool
-- = exceeds -Xmx3584m (3.5GB)

The final diagnostic step was reviewing the application source in Anypoint Exchange. The syncProductCatalogFlow Database Select connector XML confirmed the absence of any streaming attribute, and no maxFetchSize or fetchSize configuration. There was also no <error-handler> element on the flow — explaining the 21 days of silent failure. Diagnosis complete. Total time: 0.75 hours.

The fix: enabling Database connector streaming, fetchSize, and flow error handling

The fix required three changes to the lfsg-product-sync application. All three were made in Anypoint Studio and deployed as version 1.2.1.

Part 1: Enable Database connector streaming. The streaming="true" attribute was added to the db:select element in syncProductCatalogFlow. The full before/after XML:

<!-- Before (v1.2.0): -->
<db:select doc:name="DB_Select_Products" config-ref="LFSG_DB_Config">
    <db:sql>SELECT * FROM PRODUCT_CATALOG</db:sql>
</db:select>

<!-- After (v1.2.1): -->
<db:select doc:name="DB_Select_Products" config-ref="LFSG_DB_Config" streaming="true" fetchSize="1000">
    <db:sql>SELECT * FROM PRODUCT_CATALOG ORDER BY PRODUCT_ID</db:sql>
</db:select>

Part 2: Add flow-level error handling with Slack alerting. The original syncProductCatalogFlow had no <error-handler> element. A new <on-error-continue> handler was added that logs the error and sends an HTTP POST to the LFSG Operations Slack incoming webhook:

<flow name="syncProductCatalogFlow">
    <scheduler doc:name="Scheduler">
        <scheduling-strategy>
            <cron expression="0 0 2 * * ?" timeZone="America/New_York"/>
        </scheduling-strategy>
    </scheduler>

    <db:select doc:name="DB_Select_Products" config-ref="LFSG_DB_Config"
               streaming="true" fetchSize="1000">
        <db:sql>SELECT * FROM PRODUCT_CATALOG ORDER BY PRODUCT_ID</db:sql>
    </db:select>

    <foreach doc:name="ForEach_Products" collection="#[payload]">
        <http:request method="POST" doc:name="CRM_Upsert"
                      config-ref="CRM_HTTP_Config"
                      path="/api/products/upsert"/>
    </foreach>

    <error-handler>
        <on-error-continue type="ANY" logException="true"
                           doc:name="OnError_Alert">
            <logger level="ERROR" doc:name="Log_Error"
                    message="#['syncProductCatalogFlow FAILED: ' ++ error.description]"/>
            <http:request method="POST" doc:name="Slack_Alert"
                          config-ref="Slack_HTTP_Config"
                          path="/services/T00000/B00000/XXXXXXXXXXXX">
                <http:body>#[output application/json --- {
                    "text": "syncProductCatalogFlow ERROR: " ++ error.description
                           ++ " | " ++ now() as String
                }]</http:body>
            </http:request>
        </on-error-continue>
    </error-handler>
</flow>

Version 1.2.1 was deployed to CloudHub. Runtime Manager showed the application start successfully with no deployment errors.

A manual trigger of syncProductCatalogFlow was executed at 11:22 AM on September 24 to process the full 21-day backlog of product catalog updates (including all 847,293 current products, with the 12,847 previously-missing Meadowbrook Trust products among them). The manual sync completed in 12 minutes and 47 seconds. The JVM Heap Used dashboard peaked at 800MB throughout the run — a 77% reduction from the 3,582MB peak under the non-streaming configuration, and comfortably within the 3,584MB ceiling.

-- Manual sync run: Sep 24 2026, 11:22 AM – 11:35 AM
-- lfsg-product-sync v1.2.1, CloudHub 1.0, 2 vCore worker

Application log (INFO):
  [11:22:03] syncProductCatalogFlow: scheduled trigger fired (manual)
  [11:22:04] DB_Select_Products: streaming cursor opened on PRODUCT_CATALOG
  [11:22:04] ForEach_Products: processing started (847,293 items)
  [11:34:50] ForEach_Products: processing complete — 847,293 iterations
  [11:34:50] syncProductCatalogFlow: completed successfully

Runtime Manager → Dashboard → JVM Heap Used:
  11:22  312 MB  (baseline)
  11:23  487 MB  (ForEach iterating, 1,000-row batches, CRM upserts in progress)
  11:28  741 MB  (peak during mid-sync HTTP connection pool + active batch)
  11:34  798 MB  (peak — highest observed during full 847,293-row sync)
  11:35  389 MB  (sync complete, GC runs, heap drops to near-baseline)

CRM API validation (Sep 24, 12:08 PM):
  GET https://crm.lfsg.internal/api/products/count
  Response: { "total": 847293, "last_updated": "2026-09-24T11:34:50Z" }

  GET https://crm.lfsg.internal/api/products?source=meadowbrook_trust&limit=1
  Response: { "count": 12847, "products": [...] }  -- all 12,847 products now present

  CRM quoting tool: 847,293 products visible. Meadowbrook Trust products visible. ✓

The fix was complete. All 847,293 products were present in the CRM. The 12,847 products from Meadowbrook Trust that had been invisible in the quoting tool for 21 days were now visible. The nightly scheduled run executed successfully the following morning at 02:00 with no OOM events and no Slack alerts. Total engagement time: 3.5 hours.

Why non-streaming Database connectors are a recurring MuleSoft retainer problem

The Mule 4 Database connector’s streaming="false" default is not a design flaw — it is the correct behavior for the majority of integration use cases. Most Mule flows that query a database table are querying lookup tables, configuration tables, or filtered result sets where the row count is predictable and bounded. Materializing a 500-row lookup table into a List<Map<String, Object>> is fast, safe, and produces a payload that is easy to work with in DataWeave. The default is appropriate for those cases.

The problem arises when a flow is designed against a table that is expected to be small — or was small at the time of design — and the table grows over time due to business events that were not anticipated or accounted for in the integration design. A product catalog that holds 300,000 rows at the time of an integration deployment is not obviously a streaming candidate. Three acquisitions later, it holds 850,000 rows and the same integration fails catastrophically. The failure is sudden from the operations team’s perspective, but from a data volume perspective it was a predictable consequence of a gradual trend crossing a threshold.

The three properties of this failure class that make it difficult to detect without a dedicated retainer review are: the application behaves correctly for months or years before the failure; the failure is not progressive (the sync either succeeds completely or fails completely at the OOM threshold — there is no partial degradation that would attract attention gradually); and the default Mule 4 configuration provides no heap-usage feedback during normal operation (there is no log message warning that the Database connector is loading a “large” result set). A MuleSoft developer retainer who reviews CloudHub worker heap trends quarterly — specifically looking for flows where the heap usage at peak corresponds to a growing database table — catches this class of issue before it reaches the OOM threshold. The LFSG case would have been caught in a Q2 2026 review: the June acquisition load would have pushed peak heap usage to approximately 1.8GB, a visible inflection point on the dashboard well below the 3.5GB ceiling but high enough to flag the trend.

How HourTab helps MuleSoft developer retainers

The Lakewood Financial Services Group engagement totaled 3.5 hours: diagnosis (0.75h — 0.25h reviewing CloudHub application logs for the lfsg-product-sync application covering September 3 through September 23, identifying the nightly java.lang.OutOfMemoryError: Java heap space at 02:03:47 with the ArrayList.add at ResultSetIterator stack trace, confirming that the same exception occurred on all 21 nights with no other error types; 0.25h reviewing the CloudHub JVM Heap Used dashboard for the same period, confirming the linear heap climb from 312MB to 3,582MB (100% of —Xmx3584m) between 02:00 and 02:03:47 each night followed by the characteristic GC drop, and correlating the heap curve shape with ArrayList materialization; 0.25h executing SELECT COUNT(*) FROM PRODUCT_CATALOG on LFSG_Operations returning 847,293 rows and cross-referencing with the January 2026 data migration report showing 293,847 rows at deployment, confirming 553,446 rows added across three acquisition ETLs, and reviewing the syncProductCatalogFlow Database connector XML in Anypoint Exchange confirming streaming attribute absent, no fetchSize, no error-handler), fix (1.5h — 0.5h adding streaming="true", fetchSize="1000", and ORDER BY PRODUCT_ID to the db:select element in Anypoint Studio; 0.5h building the on-error-continue error handler with logger and Slack HTTP Request connector, testing error handler in sandbox by temporarily injecting an error into the flow; 0.5h deploying v1.2.1 to CloudHub, monitoring startup logs, verifying application status green in Runtime Manager), and manual sync coordination and validation (0.5h — 0.25h manually triggering syncProductCatalogFlow, monitoring the 12m47s run via CloudHub logs and heap dashboard, confirming heap peak 800MB and flow completion with zero errors; 0.25h validating CRM API product count 847,293 and confirming 12,847 Meadowbrook Trust products visible in quoting tool with a CRM analyst on a brief call), plus documentation and retainer SOP update (0.75h — post-incident write-up for retainer file including root cause, timeline, fix, and prevention recommendation; retainer SOP updated to add quarterly CloudHub heap trend review for all flows containing Database Select connectors with SELECT * FROM patterns against high-growth tables).

In a HourTab retainer log, the incident session reads: “syncProductCatalogFlow OOM 21 days (Sep 3–23); Mule 4.4.0 EE CloudHub 1.0; lfsg-product-sync v1.2.0; db:select streaming=false (default); PRODUCT_CATALOG 293,847 rows Jan 2026 → 847,293 rows Sep 3 2026 (3 acquisitions: Meadowbrook Trust + Pacific Regional CU + Summit Wealth); 847,293 rows × 2.78KB/row = 2.29GB List<Map<String,Object>> materialization; JVM heap max 3,584MB exhausted at 02:03:47 nightly; OOM stack: ArrayList.add → ResultSetIterator.next → ListResultSetCloser.close; no error handler, no alerting — silent for 21 days; 12,847 Meadowbrook Trust products missing from CRM quoting tool; fix: streaming=true + fetchSize=1000 + ORDER BY PRODUCT_ID + on-error-continue Slack alert; deployed v1.2.1; manual sync: 847,293 rows in 12m47s, heap max 800MB; CRM: 847,293 products, all 12,847 missing products restored; 3.5h.” The LFSG operations director sees the exact root cause (streaming=false default, 188% table growth from acquisitions), the scope of data affected (21 days, 12,847 missing products), and the fix applied at every layer (streaming configuration, database fetch optimization, flow error handling, alerting) — with no ambiguity about how the retainer time was spent.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark — no login, no portal, no “how many hours do I have left?” emails. Free for one retainer. $9/mo for up to ten.

FAQ: MuleSoft developer retainers

See also: TIBCO developer retainer · DataStage developer retainer · Informatica developer retainer