Using vRA CloudClient REST calls to export and Import property groups and definitions

I have used the CloudClient a lot in the past to export/import machine and XaaS blueprints, resource actions, etc. but there isn't a native way to export/import property groups and property definitions. 

Recently, I've needed to move  config from the development to a production platform.  The ideal approach to this would be to use vRealize Suite Life Cycle Manager (or vRSLCM if you insist on an acronym) which would enable a more CI/CD approach for vRealize Suite products for moving config through development, test, acceptance and production including the deployment and upgrade process, as well as customer specific config such as property groups...

It it covered nicely here by @virtualhobbit.  However, it's a product which is often overlooked when  discussions around vRA start, so I needed an easy way to move property groups & definitions without vRSLCM...

As I mentioned, I used the CloudClient where I can because it's easy to get started with, something customers can get familiar with without having to understand how to build up REST API calls using curl and has a quickly understandable command syntax...

You can still use the CloudClient's REST functionality to do this...  There are plenty of blogposts covering how to connect to the CloudClient so I won't go into details on that - but using the REST client is new to some people...  First off, I tried running the below command to ensure I had the required permissions to make REST calls...  This was logged onto the source vRA instance.

vra rest get --service identity --uri tenants

Which successfully returns the tenants that I have available as part of this vRA deployment.

Next, using 'vra rest get' commands, I can return the property groups and property definitions and export these to JSON:

vra rest get --service properties-service --uri propertygroups --export c:\somewhere\pg.json --format JSON

 vra rest get --service properties-service --uri propertdefinitions --export c:\somewhere\pd.json --format JSON

This will export the existing property definitions and groups into JSON files:

JSON isn't the easiest thing to read & understand, so I tend to use http://jsoneditoronline.org/ which shows the JSON script graphically.  This is also very useful when having to re-import the property groups / definitions.

Unfortunately the 'vra rest post' command will expect a single object rather than the array which we have just exported.  As such, I used jsoneditoronline.org to split out the individual objects:

By collapsing the individual objects in the array, you can copy them out to another editor window to verify they are correctly formatted JSON, then save them to individual files...  Not ideal from a speed point of view, but this approach does ensure consistency between platforms that is essential.

From the destination vRA instance, again after logging in, I ran some 'vra rest post' commands in order to post the property groups & definitions:

vra rest post --service properties-service -uri propertygroups --format JSON --data @c:\somewhere\pg.json --headers no

I can specify to include the headers which will show the full return rather than just the success code of 201:
 
vra rest post --service properties-service -uri propertygroups --format JSON --data @c:\somewhere\pg.json --headers yes

The same applies for the property definitions as well, although below I hit a case sensitivity issue where the command didn't like '\t' in a filename path, presumably because it's a tab character... so I had to use an uppercase in the string:

vra rest post --service properties-service --uri propertydefinitions --format JSON --data @c:\somewhere\file.json  (n.b., I didn't specify headers here so it defaulted to no):

 The result...  All properties groups and definitions added to the destination vRA instance successfully.  This might seem like a lot of work for you, but I am relying quite heavily on property groups and want to ensure consistency between deployments.

Next task I am going to look at is how to use vRO and the rest client to return the JSON string from the source and run through a 'for each' loop for every object in the array of definitions / groups... but for now this is way more work to automate something than is really necessary for the time it will take to do manually.

The REST client through CloudClient can be used for pretty much anything else, not just properties...  This information is available in the API guide which is on the appliance:

https://vra.domain.local/component-registry/services/docs

Happy automating!

Issues with vRO and Active Directory only returning 1000 items

I have seen a few references to this but it's not a common problem, mainly because most people can stick to under 1000 items in an OU.  For those that cannot, the problem is this...

By default, when performing a search using any tool into AD, it will only return the first 1000 which is a default limit.  I've seen a few blog posts with application-specific approaches to this but couldn't find one for vRealize Orchestrator specifically. 

Firstly, I created an empty OU and used vRO to create 1500 groups:
The script is simple. I defined an OU as an attribute and run the script while the variable nextNumber is less than 1500.

 

It's worth pointing out the pad function.  That is really useful when a customer wants to use a custom naming convention within vRA and insists on having say 4 digits.  The pad function above will add the additional zeros on by calling the function pad for the variable longNumber...  Then I increment nextNumber by the line nextNumber++.

The result from AD is clear - 1500 groups:

  

However the vRO plugin only returns the first 1000:

There is a way of setting the plugin parameters from vRO, but this doesn't change anything... The limit is a restriction from AD.

So the answer is this...  On your domain controller (or any that vRO is targeting), use ntdsutil to change the MaxPageSize limit... The commands are below:

ntdsutil

ldap policies

connections

connect to server {your dc}

quit

set MaxPageSize to {pick a number- I went for 2000}

Commit Changes

 quit

And can be confirmed by running show values:

After rebooting AD, it returned the full 1500 results (I am not sure how necessary this is...  it's Microsoft, it felt right :) plus the plugin was not behaving beforehand although this might be due to my running the DC and vRO appliance on my laptop and starting to struggle...) 

 I tested that the limit was due to the change using ntdsutil by modifying my script to start at 1500 and run while nextNumber < 3000:

Although this hit another MS limit when I was trying to browse AD using dsa.msc (Users and Computers)

 However, it shows that the value set by ntdsutil limits what AD will return:

The problem with the above fix is this...  Most Active Directory administrators won't be happy about making changes to their domain controllers using ntdsutil without a very good reason.  In reality, having more than 1000 items in an OU is unworkable; either you have more groups than you need, or not enough granularity in your OU structure...  However I am not an authority on AD design, so I will highlight the vRO workaround I used instead!

I could add that you could deploy a read-only domain controller and that might appease your awkward AD team, although this does seem a little overkill

Workaround:  I use AD groups within configuration elements.  This provides a referable item that you can get your vRO workflows to lookup every time they run.  I like this approach because it means that when exporting code from one vRO platform to another, changes are easy and obvious (rather than having to find in code where something references a test or dev group as opposed to a production group, for example)...  What I have done below is create a configuration element with the highest item available to me (it was 2000) and exported it to a .vsoconf file:

 Then I modified this to update the value to one that is not returned by the default search limit (for me, the next available server 2001).  This is an XML string which contains the distinguished name of the object in question starting with CN for commonName (you can see CN%3D where %3D is the unicode for the equals symbol):

By the way, the text editor is Syntra Small - someone asked before...!

Then import the configuration and overwrite it:

This will allow you to store the configuration element with the new updated value which you can add to your workflows to use to call on for whatever you might need to do...

Not an ideal approach, I would strongly suggest addressing the enormous amount of groups in a single OU, but I realise that is not always possible...