How to add counters to Perfmon(Performance Monitor)

The Reliability and Performance Monitor introduced in Windows is a good built-in tool that lets you monitor and study how applications you run, affect your computer’s performance, both in real-time and by collecting log data for later analysis. In this post, we will see how to use Performance Monitor or Perfmon, as it is called, in Windows 10/8. The same of course applies to Windows 7/Vista too.

How to use Perfmon

Open the WinX menu in Windows 10 and select Run. Type perfmon.exe and hit Enter to open the Performance Monitor. In the left pane, select the User Defined node, right-click on it and select New > Data Collector Set.

Give a name to that and select Create manually (Advanced) to be able to set the parameters on your own.

Select the type of data you want to include. Click Next.

Click on Add and select the performance counters you would like to log. They could typically be a memory, processor usage, etc.

 Once adding the counters are complete, then click OK button.

Then selected counters will be displayed as shown in above screenshot.

 Next Select the Sample interval and the Units and click on Next button.

Now select the location where you want to save the data.

 How would you like to run the task? Leave it at Default – or use the Change button to select some other user. Click on Finish.

You will see the new entry in the right pane. Right-click on it and select Start.

you are done, right-click on it again and select Stop.

The data log file will be created and save in the specified location. Click on the file to view it in the Performance Monitor.

You will now be able to analyze the data for each of the counters.

While the task is running, you can always edit its properties by right-clicking on the Data Collector set in the left pane and selecting Properties.

Using Perfmon we can monitor the following counters.

1. Cpu level

-procesaor time , user time ,ideal time ,cpu interrupts

2. Memory level

- available bytes , commited bytes , page faults per sec , page faults write per sec and write per sec.

3. Disk level

- disc read time ,avg disk bytes read per sec ,avg disk bytes write per sec ,avg disk queue length and current disk queue length

4. System level

- processor queue length,context switching and threads

5. Network level

- no of connections established , connections failed , throughput,network delay

6. Server level

- requests per sec ,bytes per sec , ideal workers and active wirkers.

Note: By default Perfmon extension is .blg to save the report as CSV format just change default to camma separate.

Counters

1. JVM COUNTERS:
Process CPU Time:
Indicating the total amount of CPU time consumed by JVM.
Garbage Collector time (Garbage Collection Time):
Indicates the cumulative time spent on garbage collection and total number of innovations (invokes).
Current Heap Size:
Indicates the no of kilobytes (Kb) occupied by objects.
Free memory:
Available memory in the heap.
Garbage collector interval Time: time difference between garbage collection cause.

2. CLR (Common Library Runtime) Counters:
 Exception through per second: Indicating number of managed code exceptions
thrown per second.
 Timing GC: Time spent on garbage collection.
 .net CLR memory heap size:
 .net CLR total committed bytes
 .net CLR large object heap size

3. WEB SERVER COUNTERS:
Apache:
 CPU load: Percentage of the CPU consumed by apache server.
 Request per second: The total number of request per second served by apache.
 Bytes per second(throughput)
 Busy workers: Number of active threads serving the request.
 Ideal workers: Number of inactive threads in the apache.
IIS SERVER (INTERNET INFORMATION SERVICES):
NOTE: IIS is a web app server for .Net based application.
 Bytes sent per second.
 Bytes received for second.
 Current connections.
 Request per second disconnection ratio.
 Number of request queued.
 Number of requests rejected.
Anonymous Users:
Indicating anonymous http connections in the particular tab.


4. APPLICATION SERVER COUNTERS:

Web logic server:

 Execute the thread total count: Indicating the total number of threads assigned to
queue.
 Pending request current count: Indicates number of pending request in queue
 Queue length: Number of request in the priority queue.
 Throughput: number bytes received per second.
 Exception count: It should not cross 20.
 Connections current count:
 Transactions roll back total counts: it should not cross 5.

WEBSPHERE (IBM WAS) COUNTERS:

 Concurrent request: Number of requests are concurrently processed.
 Service time: Response time for a servlet request.
 Active count: Number of active threads in the system.
 Connection pool size: Number of threads in the pool.

5. Network Counters:
 Connection established: indicating connections success ratio.
 Connections failure: Percentage of connections failure.
 Through put:
 Network latency (delay):
 Pocket loss:

6. Disk Counters:
Disk read per second:
Disk writes per second: rate of write operations on the disk.
Average disk queue length: Number of read and write request that were queued for selected disk during the sample interval.
Disk time: percentage of the elapsed time that the selected disk was busy with serving read and write request.
Note:
Average disk queue length should not cross to for every disk.
Split I/O‘s Second: Measures the rate of I/O split due to file fragmentation.
Free space: Display the percentage of total available space.

7. Database Counters:
 We can monitor oracle 10g versions using DB stats reports above 10g.
 AWR reports.

Note:
Even DBA can generates the DB trace (or) oracle trace report to identify dead locks and full table scan.

Oracle counters (for all java based applications):
 Buffer hit ratio
 Full table scan
 Indexing
 DB time
 DB CPU time
 Hard parses and soft parses
 Top 5 time taken events(in a particular durations)
 Physical read
 Physical writes
 High CPU utilized query’s
 High memory utilized query’s
 High I/O’s utilized query’s

SQL Server Counters (.NET based applications):
Using sql profile we can monitor DB activities.
Navigations:
 Open SQL server
 Choose new option
 Choose create new profile.
Counters:
Buffer cached hit ratio.
Transactions for second.
Log cached hit ratio.
Page read per second.
Page writes per second.

SAP HANA Counters:
Using SAP HANA studio we can monitor SAP HANA.
SAP HANA built on column and row based technology.But purely work on column based.

Database side statistics analysis

To analyse the data base bottlenecks we have to generate the AWR report.

How to generate a report?
Sol:
 Select any monitor which we added.
 Right click on monitor.
 Select "reports" and select "Quick" It will display quick report pop up window.
 Select "thresholds" and select "options" according to our requirement.
 Select "general"->Graph.
 Navigate to "filter" and schedule settings.
 Select the report period.
 Select the report type (html/text/xml).
 Click on generate report file.

AWR report (Automatic Workload Repository):

 To communicate with remote server we need to install putty.
 In putty we need to provide remote server host name (or) IP address and we need to
provide remote server credentials.
 We have to enter a command {sqlplus"/as sysdba";}
 Click on enter which will prompt to sql prompt.
 In sql prompt we need to enter below one:
SQL>@$ORACLE_HOME/rdbms/admin/awrrpt.
 Click enter which will ask for "Report_type".
 Provide report type as html.
 Click enters which will ask for number of days.
 Provide number of dayClick enter.
It will show "snapshot ID's".
And ask for begin snap_IDProvide begin snap id and click enter.
And Ask for end "snap_ID".
Provide end snap_ID and click enter lt asks for report name.
Provide the report name with extensions (.htm1) click enterexit.

Database bottlenecks:


1. Low buffer hit ratio:
Process: Generated AWR report and found low buffer hit ratio which is reported 60%. BHR should be more than 95%.
Cause: Due to allocation of low memory to the buffer memory, triggers low BHR.
Recommendation: Recommended DB architect people to increase buffer memory allocation by redesigning oracle SGA.

2. Low index utilization
Low index utilization can’t monitor by AWR. With the help of DBA, we found index utilization issues and asked them to provide tuning opportunities in terms of index utilization.
Cause: Low index utilization causes delayed response time.
Recommendation: Redesign the index.

3. Full table scans
With the help of DBA, we found there are lot of full table scans.
Cause: Full table scan causes delayed response time.
Recommendation: Redesign the index and implement the index of which tables are frequently accessed.

4.High I/O operation queries

Drill downWhy queries are utilizing more CPU for any I/O operation?
How much CPU utilized by the I/O operation?
Recommendation: Recommended fine tune the queries of I/O operations.

5. High Elapsed running queries
Cause: AWR reported high elapsed running queries which causes the delayed response time.
Process: We gone through the queries which contains lot of binding variables, inner queries which causes the delayed response time.
Recommendation: Recommended DB people to rewrite the query and tuning the inner queries and binding variables.

6. High CPU utilized queries
Cause: AWR reported high CPU utilized queries which causes the delayed response time.
Process: Once we gone through the AWR report which contains more number of high CPU utilized queries, we segregated that into DB level and machine level.
Recommendation: Recommended DB people to tune the query which causes high CPU utilization.

7.DB time and DB CPU time
Cause: AWR reported high DB time and DB CPU time which causes the delayed response time. It should be minimal.
Recommendation: Recommended DB people to redesign the DB architecture.

Configuration side statistics analysis

OACore issue: Oacore process the requests and consumes the memory from JVM heap.
Scenario 1: I have an application called oracle R12/oracle apps/Ebiz which has to support 300 users, expected response times are below 5 seconds. While ramping up the users, application performance is degraded at 100 user load. Even application itself crashed at 150 user load.

Process: Initially we started analyzing client side statistics and found that issue with server side. Then we analyzed OS/hardware/code/DB but we did not find anything. So moved to DB layer,
finally we found an issue with JVM settings which is a oacore settings (memory management configuration).
After analyzing oacore settings, we recommended them to tweak that setting from oacore = 1 to oacore = 3 which resolved the issue.

 Cursor Limit Issue:
We have an application which has to support 500 users
Solution: We started test with 500 users but application got crashed at 150 user load and application performance degraded at 100 user load.
We have started analyzing client side, OS, hardware, memory, method, level statistics which didn’t provide any clue. Then we started analyzing configuration settings in HTTPD file in Tomcat Apache server and database statistics. Then moved to database configuration settings.
We found an issue with cursor limit which is causing application crash.
Process: We used dynatrace profiling tool to monitor all the layers of application including configuration settings as part of testing while ramping up users, cursor limit usage reported 100% at 120 user load. Then we asked DB team to provide the cursor limit setting which is reported 50. By default cursor limit is 50 which can accommodate only 100 users. We
recommended them to change 50 to 2000 as per oracle SGA, they can tweak to 1000 for that application. After deploying new build with 1000 cursor limit supported 500 users without any performance degradation.

Thread limit:
We found an issue server reached the max number of simultaneous connections in server log.
Cause: Not enough threads in Apache.
Statistics: Due to the connection pool settings, application is not able to handle simultaneous connections created by 150 users.
Recommendations: Increase the connection pool settings from 10 to 16.

Private Byte Issue:
Process: In one of my SAP application, users are causing memory dumps as per SAP
architecture, they configured private bytes 4mb memory. If the users are trying to retrieve more than 4mb memory, then the users will be pushed to run mode to debug mode. If SAP system allocates more memory, then the user will come back to run mode. To execute the report if the
system is not allocating the memory, the users will be pushed to error mode which causes the memory dump.
Recommendation: Recommended SAP people to reconfigure private byte settings.


DOP settings and many settings will be there.
Based on scenario and as per our requirement we'll took the changes from configuration side.

Application side statistics analysis

From application side analysis we have monitor code level issues like threads, classes , packages and memory leakages etc.....
To monitor the code level issues we may use jconsole,JVM ,JMC ,dynatrace and appdynamics .


JVisualVM:
JVisualVM is a default profiling tool for JVM. To leverage the services we have to install JDK.
Steps to invoke JVisualVM:
 My Computer
 Program Files
 Java
 bin
 JVisualVM

We can Monitor
1. Local Machine
2. Remote Machine
Steps to invoke Remote Machine:
 Go to Remote tab
 Add ProcessID
 Add JMX connections by providing Port Numbers

1. Overview:
Here we can view how much memory?, XMS & XMX settins, JVM version, JRE VersionHere we can check JVM Arguments & System Properties

2. Monitor:
By default we can View CPU,Memory,Classes,threds graphs
Here we can perform GC and we take heap dump .Heap dump extension is “.hprof”
Here we can check memory leakge with the help of heap metaspace graph.

3. Threads:
Here we can view thread status like Running(), Sleep(), Park(), Waiting()
And we can take thread dump for Analysis Purpose.
Note 1: For thread dump analysis copy the thread dump and paste it in any online thread analyzer tool.
Note 2: We can’t copy Thread dump from remote machine to local machine and vice versa.

4. Sampler:
Here we can view how much CPU & Memory Utilized by each & every thread as part of JVisualVM

Thread Dump:
Whenever application is not performing well, we used to analyze thread dump.
Thread Dump is a snapshot of thread status.

There are two types of threads:
1. Demon Threads: Which are invoked by OS and hardware level
2. Non Demon Threads: Which are created by program.

Thread contention:
Thread contention is a status in which one thread is waiting for a lock which is locked by some other thread.

Deadlock:
Deadlock is a situation where one or more threads are waiting for other resources which are locked by some other threads.

Thread synchronization:
This code will allow the threads to use multiple resources by multiple threads.
In Java, every object has one monitor. At any point of time, only one thread get lock on monitor. Other threads will wait until monitor will be released.
Note: Thread dumps will be published with the help of jstack or jvisualvm.
As part of analysis, we are going to identify which threads are in blocked status and waiting status.
If you find any blocked threads, verify for which stack it is looking for, who locked on expected stack, copy these details and post to developer.
If you find waiting threads, we have to find which methods are executing for particular thread (wait (), park (), sleep ()) and copy details and post to developer.

If multiple threads are trying to get locks on stacks which is locked by some other threads causes deadlock threads. Apart from above analysis, we do have internal thread analyzer tools.
Using these tools, we are deducting the deadlocks and thread level issues. For infrastructure level, we have to take CPU sampling, which thread is utilizing more CPU and report the same.


Memory Dump:
Memory dump is a snapshot of memory utilization statistics in a particular time period.
Whenever you received OOM (Out of memory) exception or memory leakages we have to take
the memory dump to find out the root cause.

We do have two types of memories
1. Stack memory: Static variables will be loaded into stack memory
2. Heap memory: Dynamic variables will be loaded into heap memory.
Memory dump contains below information:
 Objects
 Classes
 Class Names
 Class loader information
 Fields
 Primitive fields
 Garbage collection roots
 Thread level data
 Stacks
To understand the memory dump, we should drill down how many threads required for our application, how much memory required by each and every thread will give detailed idea about memory footprint (static)
Memory dump will give static memory and dynamic memory information (user session) for analysis purpose.
Using memory dump, we can identify which object, class are running for more time, we can copy them and send to the developer.
If you are not able to analyze the memory dump manually, we can use memory dump analyzertools to identify the issue.
Memory dump extension is “.phd” or “.hprof”

Note: Using Dynatrace we can drilled exact place which causing for delay in response times.
But for dynatrace license is required. 

Server side Statistics analysis

Monitoring CPU, Memory, and Disk Utilizations is called server side analysis.
Note: For all windows OS based machines CPU, memory utilization should not cross 80%. For all UNIX, Linux, Red hat OS based machines should not cross 90%.

To monitor these we may use default graphs like windows resources(for windows machines) , UNIX resources available with controller.
✓ For monitoring the windows machines we can use Perfmon tool which is available in windows system default.
Using this we can monitor local as well as remote machine but moth machines should be windows operated.
✓For monitoring the UNIX machines we can use UNIX commends like top,iostat,
vmstat,netstat,nmon.
Note : using windows resources and UNIX resources we can monitor the counters by configuring local and remote machines.

Network side statistics analysis

Client side statistics analysis:

To analyse the client side statistics we have to merge the following graphs inorder to identify the bottleneck.

 Running Vusers.
 Connections.
 Hits per seconds.
 Throughput.
 Error per second.
 Response time.

Merging: We can merge the graph in 3 ways.
Overlay Graph
Tile graph
Correlate graph

Scenario1:
Relation between Hits per second and throughput.
 Both should be directlyproportional, if not
Cause1: That could be a network bandwidth issue.
Cause2: Web server might have issue.
Cause3: Application itself having the issue.

Note: Hits are increasing and throughputs are not increasing due to the application issues. We are receiving exception page which impacts high hits low throughput.

Scenario2:
Relation between running users and hits per second?
 Both should be directly propositional, if not application itself having a problem (or)
application is not responding well.

Scenario3:
Relation between throughput and response time.
 Both should be inversely proportional.
Note: As per the market standard both should be inversely proportional based on boundaries (if you are testing pages).

Scenario4:
Relation between running users and connections
 Both should be directly proportional, if not
Cause1: Connection limit issues in the web server.
Cause2: Number of treads limit reached threshold point in the web server.
Controller output message for above issue:
1. Users permanently or prematurely shutdown
2. Web server log max client error.


WEB PAGE DIAGNOSTICS:
Using Web page diagnostics graph we drill down the issues related to component, network, and server level issues (break down drill).

Component Break down graph:
Which will allow you to analyze component level issue any one of the component getting delay to download which will reported as an issue.

Time Taken For First Buffer Graph (TTFB):
If the TTFB is high then the problem is with server or application.
If the TTFB is low and the page response time is very high then that is a network issue.

Note: After this we have to move to server side analysis.