Planning and Designing a Custom Data Exchange

Overview

Large custom Data Exchanges can sometimes show a Failed status even though related imports complete successfully. This behavior typically occurs when the parent Data Exchange exceeds the maximum time allowed for the initial request while child Data Exchanges that were already started continue processing independently.

Understanding how request timeouts, child Data Exchanges, and processing models work together helps Administrators and Partners diagnose execution status discrepancies and design integrations that remain reliable as data volumes grow.

A large custom Data Exchange can take longer than the system waits for its initial response. When that wait period expires, the parent Data Exchange can be marked Failed even though related imports already started may continue processing successfully.

Before building a custom integration, Partners should determine whether the initial request:

Waits for the final outcome before returning a response (synchronous processing), or
Returns immediately while work continues in the background (background processing)

A custom Data Exchange can perform a simple transformation or manage a complex integration that calls external systems, creates multiple batches, and triggers additional Data Exchanges. Long-running integrations are more likely to encounter request timeout limitations if processing remains tied to a single request.

This article provides planning guidance for custom Data Exchanges, outlines the conditions that lead to timeout-related failures, and documents the recommended architecture for long-running integrations.

Common Scenario: A Long-Running Custom Data Exchange

Key Terms

Term	Description
Data Exchange	Transfers or processes data between systems.
Custom Processor	Contains the integration-specific instructions.
Parent Exchange	Starts related child exchanges, such as separate imports for categories and products.
Child Exchange	A Data Exchange started by another Data Exchange.
Batch	A group of records processed together.
PIM	Product Information Management system used to manage product information.
ERP	Enterprise Resource Planning system used to manage business operations.
API	An interface that allows systems to communicate.
Timeout	The point at which a system stops waiting for a response.

Consider a custom product Data Exchange that works with small data sets but fails after several minutes (for example, approximately two minutes) on every large run. Some of the child imports that were started continue and complete successfully.

A typical custom processor of this kind performs the following actions during a run:

Reads products from an external PIM page by page.
Evaluates and groups products by their attributes.
Builds batches of records, for example 10,000 records per batch.
Triggers separate child Data Exchanges for attributes, categories, and products.
Waits a configured delay between batches, for example approximately 18 seconds.

Typical Symptoms

Small runs succeed
The processor completes successfully during local testing and with small batches.
Large runs fail
The parent Data Exchange is marked Failed with a request timed out error from the Commerce Connector client.
Statuses disagree
Child Data Exchanges already started continue running and are recorded as Completed Successfully after the parent has failed.
Increasing timeout values does not resolve the issue
Raising a single timeout setting does not make large runs reliable.

Root Cause

In this scenario, the custom processor performs all processing inside the single request that started it. As a result, the run can only continue for as long as the shortest timeout in that request path allows.

When a Data Exchange runs a custom processor:

Commerce Connector creates an execution log with a status of Started.
Commerce Connector calls the Custom API execute-data-exchange endpoint.
Commerce Connector keeps the request open until the processor returns.
If the processor returns false, throws an error, or the request times out, the run is recorded as Failed.

The request passes through several layers, and each layer can have its own timeout:

Calling client
Commerce Connector
API Gateway
Proxy, ingress, or load balancer
Custom API
External PIM, ERP, or another source system

As data volume grows, the time spent on source API calls, batching, child imports, and configured delays grows as well. Once processing exceeds the shortest timeout in the request path, the request ends and the parent exchange is marked Failed.

Child Data Exchanges that were already started are independent requests and continue to run to completion.

Increasing one timeout does not guarantee that other layers continue waiting for the same duration. Even if all timeout values are increased, processing time continues to grow with data volume, meaning the limit is moved rather than removed.

How to Resolve It

Move processing to a background task so the request returns promptly and the processor reports the final status after processing completes.

The implementation consists of three parts.

1. Return Promptly from ProcessData()

In this approach, the processor starts the work using Task.Run and returns true rather than waiting for the integration to finish.

Validate required configuration before starting the background task.

This shortens the execute-data-exchange request but does not guarantee a response within milliseconds or eliminate every possible timeout scenario.

public Task<dynamic> ProcessData()
{
    // Allow only one run at a time on this Custom API instance
    if (Interlocked.CompareExchange(ref _runInProgress, 1, 0) != 0)
        return Task.FromResult<dynamic>(false);

    IServiceScopeFactory scopeFactory =
        _serviceProvider.GetRequiredService<IServiceScopeFactory>();

    _ = Task.Run(() => RunInBackground(scopeFactory));

    return Task.FromResult<dynamic>(true);
}

2. Give the Background Work Its Own Service Lifetime

The task creates a new dependency injection scope (DI scope), which manages the services used during that task.

The background task resolves its own IHandlerClient and configuration from that scope.

Services associated with the initial request are disposed when the request ends. The background task must not continue using those services after ProcessData() returns.

private async Task RunInBackground(IServiceScopeFactory scopeFactory)
{
    using IServiceScope scope = scopeFactory.CreateScope();

    bool success = false;
    string message;

    try
    {
        message = await Run(scope.ServiceProvider);
        success = true;
    }
    catch (Exception ex)
    {
        message = "Processing failed: " + ex.Message;
    }
    finally
    {
        Interlocked.Exchange(ref _runInProgress, 0);
    }

    await ReportFinalStatus(success, message);
}

3. Report the Final Status Once

When processing completes, the processor calls the Update Process Log Counters and Status API using the logId that Commerce Connector passed to the execute-data-exchange endpoint.

While processing continues, the parent Data Exchange remains in the Started status.

API Example

PUT /dataexchanges/custom-data-exchange/{erpExchangeCode}/update-processor-log/{logId}

{
  "Status": true,
  "SuccessRecordCount": 50000,
  "FailedRecordCount": 0,
  "LogMessage": "All batches imported"
}

Important API Behavior

If Status is omitted, the run is recorded as Failed.
The counts replace the values stored on the log and are not added to them.
Every call sets the end date of the run.
The API should be called only once when processing completes.
Failed runs should also call the API with:
- Status set to false
- A meaningful log message
- Details identifying the failed processing step

Validation Example

A representative test was performed using:

50,000 products
10,000-record batches
Approximately 18 seconds between batches

With background processing:

The initial request returned immediately.
The run was no longer interrupted by request timeout limitations.
Processing continued independently until completion.

Planning a Custom Data Exchange

Before building a custom processor, define the use case, estimate the workload, and select a processing model that can complete reliably.

1. Understand the Use Case

Identify:

The source system and destination system.
The type of data being exchanged.
Whether the process is a full synchronization or incremental update.
How frequently the exchange runs.
Whether processing steps must run in a specific order.
Whether the processor triggers additional Data Exchanges.
Whether the caller requires the final result in the initial response.
What constitutes a successful outcome.
What should occur if one step fails.

2. Estimate the Workload

Record count alone does not determine processing time.

A smaller number of records containing numerous attributes can require more processing than a larger number of simpler records.

Estimate:

Expected record count.
Peak record count.
Anticipated growth.
Payload size.
Number of attributes per record.
Number of source API calls and expected response times.
Batch size.
Number of batches.
Number of downstream Data Exchanges.
Mapping and transformation complexity.
Configured delays between batches.
Expected total processing time.

Important: Validate batch sizes with representative data rather than record count alone.

3. Choose the Processing Model

	Synchronous Processing	Background Processing
How it works	The request remains open until the processor returns the final result.	The request starts the work and returns immediately. The processor reports the final status later.
Recommended for	Short, predictable workloads with controlled volume.	Large or variable workloads with multiple API calls, batches, delays, or child exchanges.
Total run time	Must fit within the shortest timeout in the request path.	Not limited by request timeout.
Initial response	Returns the final result.	Returns confirmation that processing started.
Final status set by	Commerce Connector based on the processor return value.	The processor using the Update Process Log Counters and Status API.

Background processing is not required for every custom processor.

Background processing is recommended when the workload cannot reliably complete within a single request, either today or as data volume increases in the future.

A successful initial response confirms only that processing started successfully. It does not confirm that the Data Exchange completed successfully.

Background Processing in Practice

The parent Data Exchange should finish with the status of the entire workflow rather than the status of the first request.

Example Parent-Child Workflow

In this example:

ParentProductExchange is the primary product integration.
ExternalPimSource provides product information.
ProductAttributeImport imports product attributes.
CategoryImport imports categories.
ProductImport imports products.

The first request finishes as soon as the background process starts. The parent's final status is recorded only after all required child exchanges complete.

Commerce Connector creates the parent execution log with a status of Started.
The custom processor validates the request and required configuration.
The processor starts the background work and the initial request returns.
The parent exchange remains in Started while processing continues.
Each child exchange follows its normal Commerce Connector processing flow and receives its own execution log and final status.
If all required child exchanges succeed, the processor updates the parent to Completed Successfully.
If a required child exchange fails, the processor updates the parent to Failed and identifies the failed step.

Implementation Checklist

ProcessData() validates configuration, starts processing with Task.Run, and immediately returns true.
The background task creates its own DI scope.
The background task does not use services from the original request.
Only one processor execution can run at a time on each Custom API instance.
Delays between batches can be cancelled during Custom API shutdown rather than using a blocking Thread.Sleep.
Every outcome, including exceptions and shutdown events, ends with a single call to the Update Process Log Counters and Status API.
The final call includes:
- Status
- SuccessRecordCount
- FailedRecordCount
- LogMessage
The failed step is identified when applicable.
Final status updates are retried on network or server failures so temporary outages do not leave the parent exchange in a Started status.

IMPORTANT:

Before You Begin: Plan the Processing Model

A custom processor runs inside the request that Commerce Connector sends to the Custom API execute-data-exchange endpoint.

If the processor does not return before the shortest timeout in that request path, the Data Exchange is marked Failed, even if processing that was already started continues to completion.

Before building a custom processor, estimate the workload:

Record count
Payload size
External API calls
Batches
Processing delays
Downstream Data Exchanges

If the workload may not reliably complete within a single request, use background processing:

Start the work.
Return the initial response immediately.
Record the final result using the Update Process Log Counters and Status API.