Optimizing SAM Search & SAM Processing Related Workflows

Optimizing SAM Search & SAM Processing Related Workflows 
In current large production instance system is taking lot of time during the SAM assignments process and that's making entire flow very slow here.
we can see there are total 2000922 SAM records present for one organization and filtering this is becoming huge task here.

Case is Customer has 1 service agreement with multiple organization , location , request class combination , out of which one organization has many SAM associated to it

With the current design and this data setup we cannot achieve good performance , we have to either optimize the logic or the data .

in future with more request classes added this count will increase and difficult to control the capacity with this huge number.

Technically the retrieve task ( Get SAM for Request Class and Organization) of the
WF# triTask - Synchronous - SAM - Proximity Search - Organization

itself is taking 14minutes here and is the cause here .
As there are many association and this task is taking more time to complete

In total making the work  task creation to activation time between 14 to 17 minutes

Issue is happening only when the Single customer/org has many SAM records and this filtering is taking a lot time here. In Normal Dev / test instances we don't have this number and its running very fast.
 

Upon identifying we found that starting from the SAM Search process which starts from below WF, 
triTask - Synchronous - SAM Proximity Search
 

1. There are many redundant/ duplicate workflow tasks in the process ,  we are retrieving the same Request Class  / Location  multiple times , which can be optimized by passing it as variable from one wf to another . 

Another observation is retrieve task to retrieve Request class , SAM records using associations is taking more time , and when this was replaced with Query task then we had seen  very good results in retrieving them.

2. Replacing retrieve tasks ( like Get SAM for Request Class and Organization ) with query tasks in the WF will help to run the process faster.

3. If possible DB indexing on service matrix table to run the queries faster