Using the Pure Storage PowerShellSDK2 - Part 3 - Getting Performance Data from FlashArray
Welcome back to the third installment of our blog series on using the Pure Storage PowerShell SDK2. In this post, we’ll learn how to retrieve performance data from FlashArray and Cloud Block Store. Here, you’ll uncover the intricacies of extracting performance data across several object types, including Volumes and Hosts. We will dig into the object model that exposes crucial performance insights. Moreover, we’ll delve into the realm of performance analysis, addressing common customer questions such as:
- Identifying the top 10 volumes in terms of IOPS or latency.
- Pinpointing the top 10 hosts in terms of IOPS or latency.
- Finding the top 10 volumes in terms of IOPs during specific time intervals in the past.
While Pure1, the Purity web interface, and our Open Metrics Exporter for FlashArray our platforms are capable of providing this information effortlessly in several ways. This post will empower you with the skills to retrieve and extract this performance data using the Pure Storage PowerShell SDK2. With this, you can seamlessly perform ad hoc analysis or integrate this data into other analytical tools to better understand your storage performance.
Finding the Performance cmdlets
We will start with discovering which cmdlets expose performance data to us and for which objects in the array. If we look at the cmdlets with the string performance in the name, we can see what’s available to us to work with. In the listing below, you can see several object types, Array, Directory, Host Group, Host, NetworkInterface, Pod, Volume Group, and Volume.
This post focuses on Volumes and Hosts. We can answer those common customer questions we mentioned in the previous section. We can also use the techniques in this post to analyze performance against these other object types since the performance information exposed is nearly the same for all objects.
Get-Command -Module PureStoragePowerShellSDK2 | Where-Object { $_.Name -like "*performance" }
CommandType Name Version Source
----------- ---- ------- ------
Cmdlet Get-Pfa2ArrayPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2DirectoryPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2HostGroupPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2HostPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2NetworkInterfacePerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2PodPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2VolumeGroupPerformance 2.24.70 PureStoragePowerShellSDK2
Cmdlet Get-Pfa2VolumePerformance 2.24.70 PureStoragePowerShellSDK2
Understanding the Resource Performance Object Model
First, I want to show you what performance information you can retrieve from FlashArray or Cloud Block Store via the PowerShell SDK2 module. In the last section, I showed you some cmdlets that exposed performance information for the many objects available in FlashArray and Cloud Block Store. In this section, let’s look at the cmdlet that exposes Volume information with Get-Pfa2VolumePerformance. In the output below, you’ll find that the type is PureStorage.FlashArray.Rest.Model.ResourcePerformance, and you can see performance information across many dimensions exposed as Properties.
Get-Pfa2VolumePerformance -Array $FlashArray | Get-Member
TypeName: PureStorage.FlashArray.Rest.Model.ResourcePerformance
Name MemberType Definition
---- ---------- ----------
BaseToString Method string BaseToString(string[] names, System.Object[] values)
CompareMembers Method bool CompareMembers(System.Object lhs, System.Object rhs)
Equals Method bool Equals(System.Object input), bool Equals(PureStorage.FlashArray.Rest.Model.ResourcePerformance input), bool IEquatable[ResourcePerformance].Equals(PureStorage.FlashArray.Rest.Model.ResourcePerformance other)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
GetValue Method long GetValue(System.Object value)
HashMembers Method int HashMembers(int hashCode, int value), int HashMembers(int hashCode, long value), int HashMembers[T](int hashCode, T value)
ToString Method string ToString()
BytesPerMirroredWrite Property System.Nullable[long] BytesPerMirroredWrite {get;set;}
BytesPerOp Property System.Nullable[long] BytesPerOp {get;set;}
BytesPerRead Property System.Nullable[long] BytesPerRead {get;set;}
BytesPerWrite Property System.Nullable[long] BytesPerWrite {get;set;}
Id Property string Id {get;}
MirroredWriteBytesPerSec Property System.Nullable[long] MirroredWriteBytesPerSec {get;set;}
MirroredWritesPerSec Property System.Nullable[long] MirroredWritesPerSec {get;set;}
Name Property string Name {get;set;}
QosRateLimitUsecPerMirroredWriteOp Property System.Nullable[long] QosRateLimitUsecPerMirroredWriteOp {get;set;}
QosRateLimitUsecPerReadOp Property System.Nullable[long] QosRateLimitUsecPerReadOp {get;set;}
QosRateLimitUsecPerWriteOp Property System.Nullable[long] QosRateLimitUsecPerWriteOp {get;set;}
QueueUsecPerMirroredWriteOp Property System.Nullable[long] QueueUsecPerMirroredWriteOp {get;set;}
QueueUsecPerReadOp Property System.Nullable[long] QueueUsecPerReadOp {get;set;}
QueueUsecPerWriteOp Property System.Nullable[long] QueueUsecPerWriteOp {get;set;}
ReadBytesPerSec Property System.Nullable[long] ReadBytesPerSec {get;set;}
ReadsPerSec Property System.Nullable[long] ReadsPerSec {get;set;}
SanUsecPerMirroredWriteOp Property System.Nullable[long] SanUsecPerMirroredWriteOp {get;set;}
SanUsecPerReadOp Property System.Nullable[long] SanUsecPerReadOp {get;set;}
SanUsecPerWriteOp Property System.Nullable[long] SanUsecPerWriteOp {get;set;}
ServiceUsecPerMirroredWriteOp Property System.Nullable[long] ServiceUsecPerMirroredWriteOp {get;set;}
ServiceUsecPerReadOp Property System.Nullable[long] ServiceUsecPerReadOp {get;set;}
ServiceUsecPerReadOpCacheReduction Property System.Nullable[float] ServiceUsecPerReadOpCacheReduction {get;set;}
ServiceUsecPerWriteOp Property System.Nullable[long] ServiceUsecPerWriteOp {get;set;}
Time Property System.Nullable[datetime] Time {get;set;}
UsecPerMirroredWriteOp Property System.Nullable[long] UsecPerMirroredWriteOp {get;set;}
UsecPerReadOp Property System.Nullable[long] UsecPerReadOp {get;set;}
UsecPerWriteOp Property System.Nullable[long] UsecPerWriteOp {get;set;}
WriteBytesPerSec Property System.Nullable[long] WriteBytesPerSec {get;set;}
WritesPerSec Property System.Nullable[long] WritesPerSec {get;set;}
Tip: Examine the other object types’ (
Array,Hosts,Host Groups, and more) and their properties using the performance cmdlets we discovered previous section and piping that intoGet-Member.
Deep Dive on the Performance Metrics
Now, there are a lot of performance metrics exposed here. Below is a table summarizing the performance metrics interesting to this blog post with a description. There are additional metrics for advanced scenarios like ActiveCluster and user-defined QoS policies.
There are three columns in this table. The PowerShell property is the actual property name the PowerShell object returns. The API Property is the actual REST API property. There is also a short description of what the performance metrics mean. The PowerShell object property is camelcase, and the array API response property has a different format, separating parts of the name with underscores _. For example, ReadsPerSec is the PowerShell property, and reads_per_sec is the array API response property. Notice the underscores are removed from the PowerShell property, and the API response property is lowercase and is case sensitive.
As discussed in our previous post Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data when working with data on the client side you will use the PowerShell Property. When working with data inside the REST API, you will use the API Property.
TIP: If you’re having trouble finding the right API property, visit the FlashArray REST API Reference Guides.
| PowerShell Property | API Property | Description |
|---|---|---|
| BytesPerOp | bytes_per_op | Average I/O size measured in bytes, both read and write |
| BytesPerRead | bytes_per_read | Average read I/O size measured in bytes |
| BytesPerWrite | bytes_per_write | Average write I/O size measured in bytes |
| Id | id | A unique identifier for the object |
| Name | name | The user friendly name of the object |
| QueueUsecPerReadOp | queue_usec_per_read_op | Average time that a read I/O request spends in the array waiting to be served, measured in microseconds |
| QueueUsecPerWriteOp | queue_usec_per_write_op | Average time that a write IO request spends in the array waiting to be served, measured in microseconds |
| ReadBytesPerSec | read_bytes_per_sec | The number of bytes read per second |
| ReadsPerSec | reads_per_sec | Number of read requests processed per second, read IOPs |
| SanUsecPerReadOp | san_usec_per_read_op | Time required transferring data between initiator and FlashArray for read IOs, measured in microseconds |
| SanUsecPerWriteOp | san_usec_per_write_op | Time required transferring data between initiator and FlashArray for write IOs, measured in microseconds |
| ServiceUsecPerReadOp | service_usec_per_read_op | The average time required for the array to service a read request. Measured in microseconds. |
| ServiceUsecPerWriteOp | service_usec_per_write_op | The average time required for the array to service a write request. Measured in microseconds. |
| Time | time | The time when the sample performance data was taken. Measured in milliseconds since the UNIX epoch. |
| UsecPerReadOp | usec_per_read_op | The average time it takes the array to process an I/O read request, measured in microseconds. The average time does not include SAN time, queue time, or QoS rate limit time. |
| UsecPerWriteOp | usec_per_write_op | The average time it takes the array to process an I/O write request, measured in microseconds. The average time does not include SAN time, queue time, or QoS rate limit time. |
| WriteBytesPerSec | write_bytes_per_sec | Amount of data written per second |
| WritesPerSec | writes_per_sec | Number of write requests processed per second |
Table 1: Performance Metrics Exposed via PowerShell
Tip: When working with latency metrics, latency is broken out into components measuring different latency areas in the I/O path, such as SAN Time, Service Time, Queue Time, and QoS Time. Descriptions are in the above chart. This is immensely useful for troubleshooting as you can isolate where you’re latency is occurring.
Kick Off a Workload
I will kick off a workload so we can have meaningful data to look at and analyze. Below is the code to kick off a SQL Server database backup using the dbatools PowerShell module. In your environment, you likely have workloads running. In my lab…I need to provide something interesting for us to look at here ;)
Start-Job -ScriptBlock {
$username = "sa"
$password = 'S0methingS@Str0ng!' | ConvertTo-SecureString -AsPlainText
$SqlCredential = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $password
Backup-DbaDatabase -SqlInstance 'aen-sql-22-a' -SqlCredential $SqlCredential -Database 'FT_Demo' -Type Full -CopyOnly -FilePath NUL
}
Finding Busy Volumes
Using our API sorting method discussed in our previous post Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data, in the code below I’m using Get-Pfa2VolumePerformance and the -Sort parameter to sort by reads_per_sec- descending. And then, I’m using the -Limit parameter to return only the top 10 volumes.
I’m then piping that output to Select-Object Name, Time, ReadsPerSec, BytesPerRead so that I’m returning only the name, the time of the sample, the read IOPs, and the average read IO size in bytes. Who is the top culprit in terms of read IO, that volume supporting the SQL Server database backup I kicked off earlier in the post, vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242. This volume is reading 3586 read IOs/sec with an average read IO size of around 512KB.
Get-Pfa2VolumePerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 |
Select-Object Name, Time, ReadsPerSec, BytesPerRead
Name Time ReadsPerSec BytesPerRead
---- ---- ----------- ------------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242 10/6/2023 6:30:30 PM 3586 524288
PRODDB003 10/6/2023 6:30:30 PM 920 143206
fds23123123 10/6/2023 6:30:30 PM 21 421522
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663 10/6/2023 6:30:30 PM 19 23067
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01 10/6/2023 6:30:30 PM 8 80896
vvol-catalog-18098d57-vg/Config-11b28d2b 10/6/2023 6:30:30 PM 5 614
SQLFB-REP 10/6/2023 6:30:30 PM 0 0
SQLFB-DB1 10/6/2023 6:30:30 PM 0 0
SQLFB-DB2 10/6/2023 6:30:30 PM 0 0
SQLFB-DB3 10/6/2023 6:30:30 PM 0 0
Note: The default resolution on storage objects like
Volumesis a 30-second window starting when the cmdlet is executed.
Using this powerful PowerShell technique, you can quickly sort your way through performance data across any of the performance metrics mentioned that we learned about in the previous section and also across the many different object types in FlashArray or Cloud Block Store such as Host, Host Groups and the others listed in the section.
But Where is Total IOPs?
In the object model we just went through, you see ReadsPerSec and WritesPerSec, but there isn’t a property for total IOPs. In this case, we must calculate that locally on the client side. So, in the code below, I’m taking a performance sample with Get-Pfa2VolumePerformance and storing that in the variable $VolumePerformance. From there, we can use the data in that variable in local memory to perform our calculations quickly. And that’s what we have below. I’m adding ReadsPerSec to WritesPerSec and creating a new calculated property, IOsPerSec. Once the calculation completes, that’s passed across the pipeline to Sort-Object -Property IOsPerSec -Descending to sort by the IOsPerSec, and then we use Select-Object -First 10 to limit the output to the top 10.
$VolumePerformance = Get-Pfa2VolumePerformance -Array $FlashArray
$VolumePerformance |
Select-Object Name, ReadsPerSec, WritesPerSec, @{label="IOsPerSec";expression={$_.ReadsPerSec + $_.WritesPerSec}} |
Sort-Object -Property IOsPerSec -Descending |
Select-Object -First 10
Name ReadsPerSec WritesPerSec IOsPerSec
---- ----------- ------------ ---------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242 3199 0 3199
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7 0 732 732
sn1-m70-f06-33-vc01-ds02 42 115 157
sn1-m70-f06-33-placeholder 90 9 99
sn1-m70-f06-33-vc01-ds01 12 59 71
vvol-catalog-18098d57-vg/Config-11b28d2b 15 33 48
pxclouddrive-ec24204b91 2 20 22
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01 16 0 16
vvol-MUTTDB001-bd7156db-vg/Data-fa47a22c 11 0 11
vvol-ds3e2-fsa-lab-dc2639f4-vg/Data-c853f680 0 6 6
So, there are some scenarios where you’ll need to do client-side calculations. Here we see an example around IOPS another example is performing calculations on the latency metrics. Bringing the performance data locally and performing the calculations in memory can provide excellent performance since it will minimize the number of round trips to the array. This data is a point-in-time representation of the performance data, you will need to sample the data again to get a fresh sample.
Finding Busy Hosts
So, now that you know the object model and how to work with performance data, let’s look at one more example. Let’s find the top 10 busiest hosts on the array by read IOPs. Most of what we will walk through below resembles the Volume example above. The ResourcePerformance objects are nearly identical between the different object types such as Volume and Host. Let’s jump right into the code and show you how to find the top 10 busiest hosts on your FlashArray or Cloud Block Store in terms of read IOPs.
Using the cmdlet Get-Pfa2HostPerformance, we can use FlashArray’s API to -Sort on our API property reads_per_sec and Limit the output to the top 10…we then pipe the output to Select-Object Name, Time, ReadsPerSec, BytesPerRead to select the exact properties we want and there’s the listing.
Get-Pfa2HostPerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 |
Select-Object Name, Time, ReadsPerSec, BytesPerRead
Name Time ReadsPerSec BytesPerRead
---- ---- ----------- ------------
sn1-r720-f06-09 10/6/2023 7:52:02 PM 2634 110026
sn1-r720-f06-13 10/6/2023 7:52:02 PM 1199 65556
sn1-cfi-c08-15-c01-b02 10/6/2023 7:52:02 PM 108 18849
sn1-cfi-c08-15-c01-b04 10/6/2023 7:52:02 PM 96 19456
sn1-cfi-c08-15-c01-b03 10/6/2023 7:52:02 PM 81 13754
sn1-x70-f06-27:sn1-r720-f07-03 10/6/2023 7:52:02 PM 5 78643
sn1-r720-f07-03 10/6/2023 7:52:02 PM 2 65536
node-1-1 10/6/2023 7:52:02 PM 1 16384
sn1-r720-f06-15 10/6/2023 7:52:02 PM 0 0
sn1-r720-f07-01 10/6/2023 7:52:02 PM 0 0
Let’s Learn How to Look Back in Time…
So now you know how to get performance data for different objects, such as Volume and Hosts, from FlashArray or Cloud Block Store. By default, this data is for a sample from the time of the cmdlet execution and a window of 30 seconds back. So that’s good for answering the questions regarding what’s happening in your array right now, but now let’s talk about how to look back in time so that we can answer the question of what happened when.
All of the performance cmdlets we learned about above expose three parameters to help with this
-StartTime- Displays historical performance data for the specified time window, whereStartTimeis the beginning of the time window.-EndTime- Displays historical performance data for the specified time window, whereEndTimeis the end of the time window.-Resolution- The number of milliseconds between samples of historical data.
Here, I will show you how to use a fixed time window defining both parameters StartTime and EndTime. There are some additional use cases around how one can use these parameters. Please read the help for the cmdlets to learn more.
In the code below, let’s find the top 10 busiest volumes sorted by read IOPs, looking back at a one-day window that started two days ago. When looking back in time, you must define a Resolution for your sampling. I’m using 1800000 milliseconds, which is 30 minutes.
With the date math on the first three lines, we’re defining variables $StartTime and $EndTime for the parameters -StartTime and -EndTime.
We then use Get-Pfa2VolumePerformance again and add in these three parameters, StartTime, EndTime and Resolution. What’s returned is the top 10 volumes in the array for a one-day window, starting three days ago and ending two days ago. We then pipe that into Select-Object to filter the output to the results we want to output.
$Today = (Get-Date).ToUniversalTime()
$StartTime = $Today.AddDays(-3)
$EndTime = $Today.AddDays(-2)
Get-Pfa2VolumePerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 -StartTime $StartTime -EndTime $EndTime -resolution 1800000 |
Select-Object Name, Time, ReadsPerSec
Name Time ReadsPerSec
---- ---- -----------
PRODDB003 10/3/2023 6:59:22 PM 344
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7 10/3/2023 6:59:22 PM 194
vvol-MUTTDB001-bd7156db-vg/Data-fa47a22c 10/3/2023 6:59:22 PM 160
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01 10/3/2023 6:59:22 PM 60
sn1-m70-f06-33-vc01-ds02 10/3/2023 6:59:22 PM 53
sn1-m70-f06-33-vc01-ds01 10/3/2023 6:59:22 PM 34
vvol-catalog-18098d57-vg/Config-11b28d2b 10/3/2023 6:59:22 PM 30
px-fsa-dr-01 10/3/2023 6:59:22 PM 28
vvol-MUTTDB001-bd7156db-vg/Data-e7199a87 10/3/2023 6:59:22 PM 27
SAPHANA-ActiveDR2::ryan-saphana-activedr-site2 10/3/2023 6:59:22 PM 16
We can extend this search by combining other techniques we learned in our previous post and add in a -Filter. This allows us to focus our search on a subset of objects. So rather than looking at all volumes in the array, we’re limiting our search to volume names with the string *aen-sql-22-a* in the name, which is all the volumes associated with a specific SQL Server in my lab. This enables us to track performance metrics for specific volumes in our array.
Get-Pfa2VolumePerformance -Array $FlashArray -Limit 10 -StartTime $StartTime -EndTime $EndTime -resolution 1800000 -Filter "name='*aen-sql-22-a*'" -Sort 'reads_per_sec-' |
Sort-Object ReadsPerSec -Descending |
Select-Object Name, Time, ReadsPerSec
Name Time ReadsPerSec
---- ---- -----------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663 10/3/2023 6:59:22 PM 6
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663 10/3/2023 7:29:22 PM 6
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-620d202b 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-701193f8 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-77084035 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-87ee3d7c 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Swap-80dfb009 10/3/2023 6:59:22 PM 0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8 10/3/2023 7:29:22 PM 0
Wrapping Things Up
To wrap up this third installment of my Pure Storage PowerShell SDK2 Blog Series, we’ve learned how to effectively retrieve performance data from FlashArray and Cloud Block Store using the Pure Storage PowerShell SDK2. We’ve gained insights into storage performance by exploring various object types, such as Volumes and Hosts. This knowledge allows you to conduct ad-hoc analysis and integrate the data into other analytical tools to understand your storage performance better. Finally, I’ve demonstrated how to analyze historical performance data using time-based parameters.
Pure Storage PowerShell SDK2 Blog Series
This article is part of a blog series covering the Pure Storage PowerShell SDK2. Check out the other posts in this series:
- Using the Pure Storage PowerShellSDK2 - Part 1 - Connecting to FlashArray
- Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data
- Using the Pure Storage PowerShellSDK2 - Part 3 - Getting Performance Data from FlashArray
- Using the Pure Storage PowerShellSDK2 - Part 4 - Classifying Workloads With FlashArray Tags
You can find the supporting code for this blog series at this GitHub Repo and you can watch a webinar Unlocking the Full Potential of Pure Storage with APIs which has a walk-through of all of these demos.
