TL;DR

The Application Insights was configured to send the data twice to the Log Analytics workspace. One of them was done by ARM, the other was enforced by an Azure Policy. We turned off the Azure Policy and removed all upstream diagnostics settings on all Application Insights resources. This saved 100 euros montl

On a recent project, we encountered duplicate logging in Azure Application insights.

💡 In the Classic Azure world, Log workspaces were not a thing yet. So when creating new application insights resources you should not encounter this problem.

Problem introduction scope, and context

As seen in the screenshot we suffered in the acceptance environment with duplicate exceptions, information, and dependencies. In the development environment, on the left screen, we did not experience this issue. Duplicate logging

😎 Sensitive information in the screenshots is blurred for obvious reasons.

Exclude the software error

To exclude the possibility of a software error, we executed these actions:

  1. Debugging the application and looking at the outgoing application insights tab.
  2. The Azure web app / Azure function is misconfigured.
  3. We tested web apps with a single instance. If a single instance generates duplicate logging, it’s surely not the instance count.

The cause

The Log Analytics Workspace was configured in diagnostic settings as well it was in the properties of the Application Insights. See the screenshots of the Azure portal. Diagnostic settings The screenshot below shows how the application insights resource is configured upstream. Properties Application Insights workspace setting

So when we concluded the configuration was duplicate, we asked ourselves:

  1. Why was it duplicate configured, couldn’t that have been spotted in an existing ARM template?
  2. The answer consists of multiple factors and solutions. The two main components were an ARM template and an Azure Policy.

1. The correct way - ARM > Workspace property

The correct implementation of the properties was created by an ARM template. The infrastructure as code makes sure we’re configuring the upstream Log Analytics Workspace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
    "type": "microsoft.insights/components",
    "kind": "other",
    "name": "ai-[YOUR-APPLICATION-INSIGHTS-NAME]",
    "apiVersion": "2020-02-02-preview",
    "location": "West Europe",
    "properties": {
        "Application_Type": "web",
        "ApplicationId": "ai-[YOUR-APPLICATION-INSIGHTS-NAME]",
        "WorkspaceResourceId": "law-[YOUR-LOG-ANALYTICS-WORKSPACE-NAME]"
    }
}
💡 The naming of Azure resources is done using the Azure abbreviations guide.

2. Azure Policy was enforced on ‘Diagnostic settings’

There also was an Azure policy checking that there was a diagnostic setting for sending data to the Log Analytics Workspace. Whenever the IT operations checked and enforced the Azure Policy, we would have duplicated the upstream to our Log Analytics Workspace.

We turned off this Azure Policy and removed all upstream diagnostics settings on all Application Insights resources.

Conclusion

Difference Application Insights and Log Analytics workspace

graph TD
AI1[Application Insights instance 1]
AI2[Application Insights instance 2]
LA1[Log Analytics Workspace]

LA1 <--- AI1
LA1 <--- AI2

Application Insights gives ‘insights’ into application logging, exceptions, and such. You can use the Kusto query language to fetch data intelligently from Application Insights. The Log Analytics workspace is a set of tables. For the client in this article, the data of the Application insights was forwarded to the Log Analytics workspace. The advantage of the Log Analytics workspace is to query over multiple Application insights as well as data about other resources in Azure, such as API management, application gateways, service busses, or firewalls.

In the screenshot below is seen that when you create a new Application Insights resource the Log Analytics Workspace is configured automatically. Create Application Insights workspace based

ℹ️ The Log Analytics workspace is part of the Azure Monitor component in Azure.

Cost analysis 💰

This change saved the client over €1000 monthly in Azure Log Analytic costs. The euros won’t tell you how much of the total it was. After fixing all duplicate configurations, the total costs of the Application Insights resources were reduced by roughly 40-50%. If you also are experiencing this problem, I hope this article helps. Good logging makes all developers happy.

Wrap up

Whenever you see duplicate logging in your application insights make sure the configuration is correct. Also, make sure that you’re not forcing a policy on the diagnostic settings when you configure it in the properties. Only one upstream to the Log Analytic workspace is required 😉.

References