Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data
Welcome back to the second installment of our series on using the Pure Storage PowerShell SDK2. In this post, we’ll dive into working with object data using Pure Storage PowerShell SDK2.
When it comes to manipulating data in PowerShell, the ability to effortlessly pipe objects and their associated data between cmdlets is a game-changer. However, when it comes to Pure Storage PowerShell SDK2, there’s an even more efficient way to handle this. By tapping into the REST API of your FlashArray and Cloud Block Store, you can significantly reduce the runtime of your cmdlet executions. This can be achieved by leveraging key functions such as sort, limit, and filter.
In this journey, we’ll explore how all of this works and how you can make the most of it. But before we dive in, if you want to follow along with the code examples, you can find the supporting code for this blog series at this GitHub Repo
Working With Data on the Client Side
First, let’s perform a common task using PowerShell pipeline techniques. Let’s write some PowerShell to give me the top ten volumes sorted total physical space in the FlashArray.
We now need to discover what properties are available to us to use. In this post, we are working with Volumes, so we can use Get-Pfa2Volume | Get-Member to help us discover what the Volume object is made of. In the output below, you have the object description for PureStorage.FlashArray.Rest.Model.Volume, the REST API representation of a FlashArray volume. You can see the Methods and Properties available. Properties are data we can use for operations like filtering and sorting. In this example, we will use the Space property to sort by the TotalPhysical space for Volumes.
As you learned in the first post in this series the
Spaceproperty has additional nested properties that are accessible by using the-ExpandPropertyparameter.
Get-Pfa2Volume -Array $FlashArray | Get-Member
TypeName: PureStorage.FlashArray.Rest.Model.Volume
Name MemberType Definition
---- ---------- ----------
BaseToString Method string BaseToString(string[] names, System.Object[] values)
CompareMembers Method bool CompareMembers(System.Object lhs, System.Object rhs)
Equals Method bool Equals(System.Object input), bool Equals(PureStorage.FlashArray.Rest.Model.Volume input), bool IEquatable[Volume].Equals(PureStorage.FlashArray.Rest.M…
GetHashCode Method int GetHashCode()
GetType Method type GetType()
GetValue Method long GetValue(System.Object value)
HashMembers Method int HashMembers(int hashCode, int value), int HashMembers(int hashCode, long value), int HashMembers[T](int hashCode, T value)
ToString Method string ToString()
ConnectionCount Property System.Nullable[long] ConnectionCount {get;}
Created Property System.Nullable[datetime] Created {get;}
Destroyed Property System.Nullable[bool] Destroyed {get;set;}
HostEncryptionKeyStatus Property string HostEncryptionKeyStatus {get;}
Id Property string Id {get;}
Name Property string Name {get;set;}
Pod Property PureStorage.FlashArray.Rest.Model.Reference Pod {get;set;}
Priority Property System.Nullable[int] Priority {get;}
PriorityAdjustment Property PureStorage.FlashArray.Rest.Model.PriorityAdjustment PriorityAdjustment {get;set;}
PromotionStatus Property string PromotionStatus {get;set;}
Provisioned Property System.Nullable[long] Provisioned {get;set;}
Qos Property PureStorage.FlashArray.Rest.Model.Qos Qos {get;set;}
RequestedPromotionState Property string RequestedPromotionState {get;set;}
Serial Property string Serial {get;}
Source Property PureStorage.FlashArray.Rest.Model.FixedReference Source {get;}
Space Property PureStorage.FlashArray.Rest.Model.VolumeSpaceCommon Space {get;set;}
Subtype Property string Subtype {get;}
TimeRemaining Property System.Nullable[long] TimeRemaining {get;}
VolumeGroup Property PureStorage.FlashArray.Rest.Model.Reference VolumeGroup {get;set;}
Now that you know the data model for the Volume object, let’s use that object. In the code below, you’ll see I’m using Get-Pfa2Volume -Array $FlashArray to return a listing of Volumes from our array. In my lab, this will return 874 Volume objects. We then pipe that output to Select-Object Name -ExpandProperty Space, exposing the additional properties of the Space property to the pipeline. I then use Sort-Object -Property TotalPhysical -Descending to sort the 874 volumes in descending order. Next is Select-Object -First 10, which then selects the top ten volumes from that sorted list. We finish it by piping it into Format-Table to get the console output into a table.
A lot, right? We are bringing down all the data, sorting it, and filtering the top ten on the client side. There’s a better way, and we’ll cover that next.
In PowerShell v7+, you can use
Sort-Object -Top 10parameter. In PowerShell 5.1 you will useSelect-Object -First 10
Get-Pfa2Volume -Array $FlashArray |
Select-Object Name -ExpandProperty Space |
Sort-Object -Property TotalPhysical -Descending |
Select-Object -First 10 |
Format-Table
Name DataReduction Shared Snapshots System ThinProvisioning TotalPhysical TotalProvisioned TotalReduction Unique
---- ------------- ------ --------- ------ ---------------- ------------- ---------------- -------------- ------
VEEAM-ExportLUNSnap-AC5FC11F8B3B49A000768BD4 2.985 0 0.711 1545391101658 16492674416640 10.333 1545391101658
SQLFB-REP 1.180 0 0.087 1275588030053 1649267441664 1.293 1275588030053
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01 3.422 0 0.872 798412501882 21990232555520 26.752 798412501882
vvol-aen-sql-01-8b810cff-vg/Data-52a441d4 41.235 687601497299 0.728 687614319949 21990232555520 151.384 12822650
sn1-m70-f06-33-vc01-ds01 3.249 181494340554 0.984 383787349990 43980465111040 209.066 202293009436
SQL-AG2-DATA 1.819 0 0.975 230149523736 16492674416640 71.530 230149523736
SQL-AG2-BACKUP 1.005 0 0.958 228563428444 5497558138880 24.052 228563428444
aen-sql-fci::aen-sql-fci-01-iscsi-prod-01 6.649 0 0.880 179288797005 10995116277760 55.321 179288797005
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7 16.618 94866204670 0.537 97010071485 96636764160 35.901 2143866815
PRODDB003 37.451 79838797496 0.947 87749876986 54975581388800 711.281 7911079490
Working With Data in the FlashArray REST API
Now, let’s push the heavy lifting of that data into the array, sorting again by the total physical space and limiting it to the top ten results. The FlashArray REST API implements Sort and Limit functions and exposes them to PowerShell as parameters on nearly every cmdlet in the Pure Storage PowerShell SDK2 module. This means we can use the parameters to do all the data computation in the REST API on the array side and return only the final results to the client.
I’m using Get-Pfa2Volume in the code below and adding two new parameters, -Sort and -Limit. Using -Sort, you can pass a value of what API Object property I want to sort on. You can find a listing of API Objects and Properties in our FlashArray REST API Reference Guides. Click on that URL and download the FlashArray REST API Reference PDF/HTML link.
We will use the parameter to sort on total physical space, -Sort space.total_physical-. Let’s unpack this code. We’re sorting on the space property and expanding that to the total_physical property. The minus sign - on the end tells the REST API to sort the data desceding. The default is ascending. You can infer the API Property by looking at the PowerShell object property in the previous code example. In PowerShell, the property is Space.TotalPhysical. The REST API object property is space.total_physical. The PowerShell property uses camelcase. The REST API object property is all lowercase and uses an underscore between each word. But if you’re having trouble finding the right property, visit the FlashArray REST API Reference Guides.
total_physicalis the total physical space occupied by system, shared space, volume, and snapshot data. Measured in bytes.
And here’s the final code that gives you the top 10 volumes by physical space but using sort and limit functions in the FlashArray REST API. This is equivalent to the previous code, where we did all the sorting and filtering on the client side. However, there is one key difference. It’s WAY more efficient regarding the amount of data moving between client and server, in run time, and how much data is transferred from server to client.
Get-Pfa2Volume -Array $FlashArray -Sort "space.total_physical-" -Limit 10 |
Select-Object Name -ExpandProperty Space |
Format-Table
Name DataReduction Shared Snapshots System ThinProvisioning TotalPhysical TotalProvisioned TotalReduction Unique
---- ------------- ------ --------- ------ ---------------- ------------- ---------------- -------------- ------
VEEAM-ExportLUNSnap-AC5FC11F8B3B49A000768BD4 2.985 0 0.711 1545391101658 16492674416640 10.333 1545391101658
SQLFB-REP 1.180 0 0.087 1275588030053 1649267441664 1.293 1275588030053
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01 3.419 0 0.872 799093079398 21990232555520 26.731 799093079398
vvol-aen-sql-01-8b810cff-vg/Data-52a441d4 41.244 687601497299 0.728 687614319949 21990232555520 151.418 12822650
sn1-m70-f06-33-vc01-ds01 3.256 181496390946 0.984 383753441631 43980465111040 209.075 202257050685
SQL-AG2-DATA 1.819 0 0.975 230149523736 16492674416640 71.530 230149523736
SQL-AG2-BACKUP 1.005 0 0.958 228563428444 5497558138880 24.052 228563428444
aen-sql-fci::aen-sql-fci-01-iscsi-prod-01 6.649 0 0.880 179288797005 10995116277760 55.323 179288797005
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7 17.603 94845900642 0.537 96843740171 96636764160 38.031 1997839529
PRODDB003 38.167 80011736352 0.947 86540885349 54975581388800 724.944 6529148997
Let’s Measure Some Performance
So we just learned why sorting and filtering in the FlashArray REST API is much more efficient since all the work on the data is done in the REST API on the array rather than bringing all the data down to the client and then working on the data. But how much faster is it?
We can use Measure-Command to measure the runtime for cmdlets. The output below shows that the PowerShell client-side method takes 2582.750 milliseconds or about 2.5 seconds.
Measure-Command {
Get-Pfa2Volume -Array $FlashArray |
Select-Object Name -ExpandProperty Space |
Sort-Object -Property TotalPhysical -Descending |
Select-Object -First 10 |
Format-Table
} | Select-Object TotalMilliseconds
TotalMilliseconds
-----------------
2582.750
Using Measure-Command again and sorting and limiting the data in the array’s REST API, the cmdlet returns data in 311.194 Milliseconds, roughly 3/10’s of a second. A very dramatic improvement here. This is a 9x performance difference, and that performance difference adds up fast at scale when building complex integrations on large systems.
Measure-Command {
Get-Pfa2Volume -Array $FlashArray -Sort "space.total_physical-" -Limit 10 |
Select-Object Name -ExpandProperty Space |
Format-Table
} | Select-Object TotalMilliseconds
TotalMilliseconds
-----------------
311.194
Let’s Measure Some More Performance
So far, we’ve looked at the sort and limit functions. There’s one more thing that I want to show you today: filter. Filter narrows down the results to only the response objects that satisfy the filter criteria. I’m using PowerShell’s Where-Object in the code below to search for Volumes with the string aen-sql-22 in the Volume name. This is a perfectly valid technique, but here, we’re pulling all 874 volumes down to the client side, passing them through the pipeline, and filtering. There’s a better way…using the -Filter parameter.
Get-Pfa2Volume -Array $FlashArray | Where-Object { $_.Name -like "*aen-sql-22*" } |
Select-Object Name
Name
----
vvol-aen-sql-22-b-9b9a3477-vg/Config-8419eb12
vvol-aen-sql-22-b-9b9a3477-vg/Data-91f04fdb
vvol-aen-sql-22-c-a55a37f5-vg/Config-41aa3630
vvol-aen-sql-22-c-a55a37f5-vg/Data-16053ad7
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8
...output omitted...
vvol-aen-sql-22-c-a55a37f5-vg/Data-441252f7
vvol-aen-sql-22-b-9b9a3477-vg/Data-f08e715f
vvol-aen-sql-22-c-a55a37f5-vg/Data-c8f8057c
vvol-aen-sql-22-b-9b9a3477-vg/Swap-8469cce4
vvol-aen-sql-22-c-a55a37f5-vg/Swap-7725cedb
Now, let’s push that into the FlashArray REST API and use a filter to reduce the amount of data returned to the client.
Get-Pfa2Volume -Array $FlashArray -Filter "name='*aen-sql-22*'" |
Select-Object Name
Name
----
vvol-aen-sql-22-b-9b9a3477-vg/Config-8419eb12
vvol-aen-sql-22-b-9b9a3477-vg/Data-91f04fdb
vvol-aen-sql-22-c-a55a37f5-vg/Config-41aa3630
vvol-aen-sql-22-c-a55a37f5-vg/Data-16053ad7
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8
...output omitted...
vvol-aen-sql-22-c-a55a37f5-vg/Data-441252f7
vvol-aen-sql-22-b-9b9a3477-vg/Data-f08e715f
vvol-aen-sql-22-c-a55a37f5-vg/Data-c8f8057c
vvol-aen-sql-22-b-9b9a3477-vg/Swap-8469cce4
vvol-aen-sql-22-c-a55a37f5-vg/Swap-7725cedb
Let’s compare the runtime of the two methods. When using PowerShell techniques, the run time is 2398.862 milliseconds; when filtering in the REST API on the FlashArray, the runtime is 379.925 milliseconds.
Measure-Command {
Get-Pfa2Volume -Array $FlashArray | Where-Object { $_.Name -like "*aen-sql-22*" } |
Select-Object Name
} | Select-Object TotalMilliseconds
Measure-Command {
Get-Pfa2Volume -Array $FlashArray -Filter "name='*aen-sql-22*'" |
Select-Object Name
} | Select-Object TotalMilliseconds
TotalMilliseconds
-----------------
2398.862
379.925
Wrapping Things Up
In this post, we’ve introduced you to the power of utilizing the FlashArray REST API for sorting, limiting, and filtering objects on the array side before returning results to the client. By adopting these techniques, you can enhance the performance of your integrations and automations. In our next post, we’ll delve deeper into retrieving performance data from your arrays using the same principles discussed here. Stay tuned for more insights and practical tips on maximizing the potential of the Pure Storage PowerShell SDK2.
Pure Storage PowerShell SDK2 Blog Series
This article is part of a blog series covering the Pure Storage PowerShell SDK2. Check out the other posts in this series:
- Using the Pure Storage PowerShellSDK2 - Part 1 - Connecting to FlashArray
- Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data
- Using the Pure Storage PowerShellSDK2 - Part 3 - Getting Performance Data from FlashArray
- Using the Pure Storage PowerShellSDK2 - Part 4 - Classifying Workloads With FlashArray Tags
- Using the Pure Storage PowerShellSDK2 - Part 5 - Checking Replication Status
You can find the supporting code for this blog series at this GitHub Repo and you can watch a webinar Unlocking the Full Potential of Pure Storage with APIs which has a walk-through of all of these demos.
