Switching from a Scheduled Workflow to Event Driven in Zuora

Issue: I have a workflow that is exceeding Zuora's task limits. Can I switch this workflow from scheduled to event based? For context, the workflow iterates through invoices to look for invoices with a certain rate plan; if the rate plan exists, then we set the 'Transferred to Accounting' field to 'Ignore'.

Switching from a scheduled workflow to an event-driven one can indeed improve the efficiency and responsiveness of your process. With a scheduled workflow running every hour, the system executes the workflow regardless of whether there are invoices that meet your criteria, potentially processing unnecessary data and consuming resources. An event-based workflow, on the other hand, triggers only when a specific event occurs—in this case, when an invoice is created.

Event-Based Workflow Advantages:

• Real-Time Processing: Invoices are processed immediately after creation, ensuring that your custom field is updated promptly.

• Resource Optimization: The workflow runs only when needed, reducing unnecessary system load compared to a scheduled workflow that runs at fixed intervals.

• Scalability: Event-driven workflows can handle spikes in invoice creation without the need to adjust the schedule frequency.

However, there's a caveat: Zuora's standard callout notifications cannot be filtered based on specific content within the invoice, such as a particular product rate plan. This means that the workflow would still trigger for every invoice created, not just those containing the rate plan you're interested in.

A secondary limitation, Zuora has limitations regarding callout notifications:

1. Single Callout per Event per Tenant: Zuora allows only one callout notification per event per tenant. If the "Invoice Posted" event callout is already in use, you cannot create another one for the same event.

2. Communication Profiles: Callout notifications are sent based on communication profiles assigned to customer accounts. Only accounts with the appropriate communication profile will trigger the callout.

Given these constraints, here are alternative approaches to achieve your goal of updating a custom field on an invoice when it contains a specific rate plan, without relying on an hourly scheduled workflow.

Addressing the Rate Plan Filtering Challenge

Since you want to avoid running the workflow unless the invoice contains the specific rate plan, we need to find a way to filter events before the workflow is triggered. Here are some strategies to achieve this:

While you cannot prevent the workflow from triggering for every invoice, you can add a decision point at the very beginning of the workflow to check for the specific rate plan. If the invoice doesn't contain it, the workflow can exit immediately.

Steps:

1. Start Task Configuration:

   • Ensure the "Start" task is set to accept incoming callouts.

   • Accept the Invoice ID as a parameter.

2. Retrieve Minimal Invoice Data:

   • Use a "Callout" or "Query" task to fetch only the necessary data to identify if the invoice contains the specific rate plan.

3. Decision Task:

   • Add a "Decision" task to check if the rate plan matches your criteria.

   • If Yes, proceed with updating the custom field.

   • If No, terminate the workflow to conserve resources.

4. Update Custom Field:

   • Use an "Update" task to modify the custom field on the invoice.

Benefits:

• Efficiency: Minimizes processing time by exiting early for irrelevant invoices.

• Simplicity: Utilizes existing Zuora features without additional infrastructure.

Considerations:

• Workflow Invocations: The workflow will still be invoked for every invoice, which may not be ideal if invoice volume is high.

• Resource Usage: Even minimal processing consumes some resources; monitor to ensure it's within acceptable limits.

To prevent the workflow from triggering unnecessarily, you can introduce an external middleware service or script that filters events before they reach the workflow.

Steps:

1. Set Up Middleware Service:

   • Develop a service (could be hosted on AWS Lambda, Azure Functions, etc.) that listens for Zuora's callout notifications.

2. Filter Events:

   • In the middleware, receive the invoice data from the callout.

   • Use Zuora's API to fetch the invoice details and check for the specific rate plan.

3. Trigger Workflow Conditionally:

   • If the invoice contains the rate plan, the middleware sends a request to your workflow's endpoint.

   • If not, the middleware does nothing, effectively filtering out unwanted events.

Benefits:

• Selective Triggering: The workflow runs only when necessary.

• Resource Optimization: Reduces the number of workflow invocations.

Considerations:

• Development Effort: Requires additional coding and maintenance of the middleware service.

• Latency: Slight delay introduced due to the additional step.

• Security and Compliance: Ensure that the middleware adheres to data protection regulations.

If the above options are not feasible, you might consider sticking with a scheduled workflow but optimizing it to reduce unnecessary processing.

Steps:

1. Adjust Schedule:

   • Determine an optimal schedule that balances timeliness with resource usage (e.g., every 2 hours instead of hourly).

2. Efficient Querying:

   • Modify your workflow to query only invoices created since the last run.

   • Use filters in your queries to retrieve only invoices that potentially contain the specific rate plan.

3. Batch Processing:

   • Process multiple invoices in a single workflow run to improve efficiency.

Benefits:

• Reduced Frequency: Less frequent runs compared to hourly schedules.

• Controlled Processing: You have complete control over when and how data is processed.

Considerations:

• Delay in Updates: Not real-time; there may be a lag in updating the custom field.

• Potential Misses: If invoices are created and require immediate action, they might not be processed until the next scheduled run.

Based on your requirements and considering the trade-offs, Option 1 (Implement Filtering Logic Within the Workflow) is the most practical solution. It allows for real-time processing and requires minimal changes to your existing setup.

Why Option 1?

• Simplicity: No need for additional infrastructure or complex coding.

• Efficiency: While the workflow triggers for every invoice, the early exit strategy minimizes resource consumption.

• Real-Time Updates: Your custom field is updated immediately when relevant invoices are created.

Implementation Tips for Option 1:

• Optimize Data Retrieval:

  • Use selective fields in your API calls to fetch only what's necessary.

  • Limit the number of records processed.

• Monitor Workflow Executions:

  • Keep an eye on the number of times the workflow is triggered.

  • Analyze the ratio of relevant to irrelevant invoices to assess the impact.

• Error Handling:

  • Ensure that the workflow gracefully handles exceptions.

  • Log instances where the invoice data cannot be retrieved.

While it's not possible to prevent the workflow from triggering for every invoice using Zuora's standard callout notifications, you can efficiently manage this by incorporating filtering logic within the workflow. This approach strikes a balance between maintaining real-time processing and optimizing resource usage.

Summary:

• Event-Based Workflow Improves Efficiency: It processes invoices as they are created, reducing unnecessary runs compared to scheduled workflows.

• Filtering Within the Workflow: Add early decision logic to exit the workflow if the invoice doesn't contain the specific rate plan.

• Alternative Options: Consider middleware filtering or optimizing the scheduled workflow, but these come with trade-offs in complexity or processing delays.

Next Steps:

1. Implement the Filtering Logic:

   • Update your workflow with the early exit decision task.

2. Test Thoroughly:

   • Create test invoices with and without the specific rate plan to ensure the workflow behaves as expected.

3. Monitor and Adjust:

   • After deployment, monitor the workflow's performance and make adjustments as needed.