Friday 25 January 2019

Issues with vRO and Active Directory only returning 1000 items

I have seen a few references to this but it's not a common problem, mainly because most people can stick to under 1000 items in an OU.  For those that cannot, the problem is this...

By default, when performing a search using any tool into AD, it will only return the first 1000 which is a default limit.  I've seen a few blog posts with application-specific approaches to this but couldn't find one for vRealize Orchestrator specifically. 

Firstly, I created an empty OU and used vRO to create 1500 groups:
The script is simple. I defined an OU as an attribute and run the script while the variable nextNumber is less than 1500.
It's worth pointing out the pad function.  That is really useful when a customer wants to use a custom naming convention within vRA and insists on having say 4 digits.  The pad function above will add the additional zeros on by calling the function pad for the variable longNumber...  Then I increment nextNumber by the line nextNumber++.

The result from AD is clear - 1500 groups:

  
However the vRO plugin only returns the first 1000:

There is a way of setting the plugin parameters from vRO, but this doesn't change anything... The limit is a restriction from AD.

So the answer is this...  On your domain controller (or any that vRO is targeting), use ntdsutil to change the MaxPageSize limit... The commands are below:
ntdsutil
ldap policies
connections
connect to server {your dc}
quit
set MaxPageSize to {pick a number- I went for 2000}
Commit Changes
 quit

And can be confirmed by running show values:

After rebooting AD, it returned the full 1500 results (I am not sure how necessary this is...  it's Microsoft, it felt right :) plus the plugin was not behaving beforehand although this might be due to my running the DC and vRO appliance on my laptop and starting to struggle...) 
 I tested that the limit was due to the change using ntdsutil by modifying my script to start at 1500 and run while nextNumber < 3000:
Although this hit another MS limit when I was trying to browse AD using dsa.msc (Users and Computers)
 However, it shows that the value set by ntdsutil limits what AD will return:

The problem with the above fix is this...  Most Active Directory administrators won't be happy about making changes to their domain controllers using ntdsutil without a very good reason.  In reality, having more than 1000 items in an OU is unworkable; either you have more groups than you need, or not enough granularity in your OU structure...  However I am not an authority on AD design, so I will highlight the vRO workaround I used instead!

I could add that you could deploy a read-only domain controller and that might appease your awkward AD team, although this does seem a little overkill

Workaround:  I use AD groups within configuration elements.  This provides a referable item that you can get your vRO workflows to lookup every time they run.  I like this approach because it means that when exporting code from one vRO platform to another, changes are easy and obvious (rather than having to find in code where something references a test or dev group as opposed to a production group, for example)...  What I have done below is create a configuration element with the highest item available to me (it was 2000) and exported it to a .vsoconf file:

 Then I modified this to update the value to one that is not returned by the default search limit (for me, the next available server 2001).  This is an XML string which contains the distinguished name of the object in question starting with CN for commonName (you can see CN%3D where %3D is the unicode for the equals symbol):
By the way, the text editor is Syntra Small - someone asked before...!

Then import the configuration and overwrite it:


This will allow you to store the configuration element with the new updated value which you can add to your workflows to use to call on for whatever you might need to do...
Not an ideal approach, I would strongly suggest addressing the enormous amount of groups in a single OU, but I realise that is not always possible...

2 comments:

  1. Great article! Running into this with a customer now. I'd like to check out/try your workaround, but the links to some of your images are broken.

    ReplyDelete
  2. Can you please upload the Images again. Kind of unclear what should i do. I am also facing same issue and getting only 1000 records.

    ReplyDelete