One of issues in system center Orchestrator 2012 Sp1 web services is that it returned either http 404 or 500 error in a lot of situations. You can experience the same if you create a new runbook and immediately invoke the web services requesting for this new runbook. You will also notice that this new runbook is not visible within the Orchestrator Web console. There are primarily two reasons for these errors to occur.
- Authorization Cache
Orchestrator maintains an authorization cache for folders and Runbooks. This authorization cache reflects the users with their rights on the folders or Runbooks.
This authorization cache is cleared every 10 minutes and is also maintained (older authorization entry timestamp compared to current time is deleted) every 30 minutes. These authorization cache and their schedules are maintained within Orchestrator SQL database. The authorization cache is also created if the authorization cache is empty and a request is made to the web service. This authorization cache is used by the web service to determine the current user’s rights and permissions on the runbook or folder. This authorization cache is used by the Orchestrator web console for its operations as well.
Orchestrator does not remove table entries from database while you delete Runbooks and folder or while importing Runbooks. Instead, Orchestrator conducts a soft-delete i.e. it updates a column called [deleted] with ‘1’ as value. This causes the Orchestrator database to grow over a period of time while numerous imports and deletes happen and newer Runbooks are getting created.
The steps undertaken by Orchestrator to build the authorization cache are to execute a series of SQL stored procedures to fill the Table [Microsoft.SystemCenter.Orchestrator.Internal].AuthorizationCache. For filling up this table, Orchestrator queries the Database for all the folders and its Runbooks and this is one of the reasons for the http 404 and 500 errors. This is because Orchestrator is even processing records for folders and Runbooks that are in deleted state for calculating the authorization cache.
2. Web service SqlCommand Command timeout
The default setting for Command timeout of SQLCommand used within system center orchestrator web service 30 seconds. This does not take into consideration that there could be thousands of rows related to folders and Runbooks that could be used for calculating the authorization cache.
HTTP 404 error
HTTP 404 error occurs when an invocation is made to the web service for a folder or Runbook that still has not found its way to the authorization cache. Typically, this happens with a runbook created recently and the authorization maintenance (as mentioned above) has not yet been executed.
HTTP 500 error
HTTP 500 error occurs when an invocation is made to the web service for a folder or Runbook and any one of the below is happening
- The authorization cache maintenance is in progress taking more than 30 seconds while the web service SQL Command timeout happens.
- There are no entries in the authorization cache and Orchestrator takes more than 30 seconds to build the cache thereby again triggering the SQL Command timeout.
One of the change in SC Orchestrator R2 is better management of authorization cache to reduce the http 500 error. This has been done by changing the SQL Stored procedures. Now, the stored procedures for both folder and Runbook authorization cache has a filter condition that filters out all the deleted folders and Runbooks for the calculation of authorization cache. In effect, in R2 only those Folders and Runbooks are processed that are alive and kicking within the designer.
The SQL excerpt within the stored procedures looks like below.
Select [UniqueId] from dbo.Folders where Deleted = 0
Select [UniqueId] from dbo.Runbook where Deleted = 0
In next post, I would provide a solution to troubleshoot these HTTP 404 and 500 errors.
Hope you would find this post useful!!