Novell is now a part of Micro Focus

Understanding and Using Persistent Search in Novell eDirectory

Articles and Tips: article

Jim Sermersheim
Senior Software Engineer
Novell, Inc.
jimse@novell.com

Vithalprasad Gaitonde
Software Consultant
Novell, Inc.
gvithalprasad@novell.com

01 Feb 2003


This AppNote introduces the Persistent Search feature available in Novell eDirectory 8.6.1 and later. This feature is an extension to the LDAP v3 search operation and provides a way for an LDAP client to receive updates from the server about changes to a search result set.


Topics

Persistent Search, LDAP, application development, directory-enabled applications

Products

Novell eDirectory 8.6.1

Audience

developers

Level

intermediate

Prerequisite Skills

familiarity with LDAP programming

Operating System

n/a

Tools

none

Sample Code

no

Introduction

As its name implies, the Persistent Search feature of Novell eDirectory is just a search operation that keeps going after the initial set of matching entries is returned. The reason it keeps going is what makes it useful. In a nutshell, this feature lets you perform a search, gather the results of the search operation, and then, whenever an entry in the result set is modified, you are sent a new copy of that entry.

In the past, developers have created LDAP applications that attempt to keep a client-side cache of a set of entries in the directory. Other applications need to know when a particular object-say a configuration object-changes, in order to trigger some logic, such as reconfiguring itself. These types of applications have had to do things like periodically poll the entries they were interested in and perform some kind of timestamp or attribute comparison to know if an entry has been modified since they saw it last. This puts a severe computational penalty on the LDAP application.

Persistent Search is an extension to the LDAP v3 search operation which moves the burden of checking for updates within a search result set from the client to the server. The Persistent Search control allows the client to perform a normal LDAP search operation (specifying the base DN, scope of search, search filter, and so on) and then, rather than having the server return a SearchResultDone message at the end, the operation maintains a connection so the client can be updated each time an entry in the result set changes. This allows the client to maintain a cache of the entries it is interested in, or trigger some logic whenever an update occurs.

Persistent Search has been proposed as an informational RFC in an IETF Internet Draft called "Persistent Search: An LDAP Change Notification Mechanism" (see http://www.ietf.org/proceedings/01aug/I-D/draft-ietf-ldapext-psearch-03.txt).

eDirectory 8.6.1 and later supports the Persistent Search control specified in the draft referenced above.

What's in a Persistent Search Control?

You can read all about the semantics and data for the Persistent Search control in the Internet Draft mentioned above. Here we'll try to capture the important stuff in plain English.

First a refresher. You might be asking yourself, "What is an LDAP search control?" In LDAP, a control is an extension mechanism that allows you to change the way an existing LDAP operation works. In this case, the control applies to the search operation. In addition to the normal things you specify when performing a search (like baseDN, scope, filter, and so on), this control lets you specify data that will help the server know how to process the Persistent Search.

Here's the relevant data that you can specify when performing a Persistent Search:

  • changeTypes. This lets you specify the types of changes you're interested in. For example, you may want to know whenever an entry is added or deleted, but you don't care if any of its attributes get changed. The changeTypes field is a logical OR of one or more of the following values: add (1), delete (2), modify (4) and modDN (8).

  • changeOnly. This is a Boolean field that tells the server whether or not you're interested in the initial result set. If ChangesOnly is True, the server will not return the result entries until modifications occur. If it's False, you get all the entries (as you normally do with a search operation), then entries repeatedly trickle in as they are updated. It's helpful to understand that there are really two phases to a Persistent Search operation: (1) the normal search phase, and (2) the "events" phase. The latter phase is where the server is done sending the initial result set, and is waiting for updates to happen in order to send them back to the client.

  • returnECs. ECs refers to "Entry Change Notifications" (explained below). If this is set to True, the server will send an Entry Change Notification control with each SearchResultEntry returned as a result of the changes.

The Entry Change Notification control provides additional information about the change that caused a particular entry to be returned as a part of the Persistent Search. This control is attached to each of the SearchResultEntry messages sent from the server to the client. In other words, each search entry that comes back due to an update will have this data attached to it (if you specified True in the returnECs field).

The relevant data in the Entry Change Notification control is:

  • changeType. The type of change that caused the entry to be returned. This is one of the following: add (1), delete (2), modify (4), or modDN (8).

  • PreviousDN. This is only present in the case of modDN operations and contains the DN of the entry before it was renamed and/or moved.

Advantages of Persistent Search

Why use Persistent Search in your applications? Let's talk about some profiles of LDAP applications that will find Persistent Search a compelling choice.

Cache Consistency

In an LDAP client application with high performance needs, you might want to maintain a temporary, local cache of information obtained through an LDAP search operation. To improve performance, the local cache is always consulted before sending a request to an LDAP server. A Persistent Search request where the changesOnly flags is False can be used if it is desirable to prime the cache; otherwise, changesOnly would be set to True in the request.

Caches are also used for other reasons besides performance improvement. In some cases, they arise naturally out of a particular application's design. For example, an LDAP client designed for administration of information held in LDAP servers will undoubtedly generate screen displays that show information gleaned from an LDAP server. The screen display is a cache that is active and visible until the user of the application takes some action that causes different information to be displayed. A refresh button or similar control may be provided to users to allow them to update the cached display. A Persistent Search request can be used instead by the administrative application to automatically refresh the screen display as soon as the underlying LDAP information changes. In fact, this is really a marriage of caching and a triggered action (described next).

Triggered Actions

An LDAP client application may want to take some action when an entry in the directory is changed. A Persistent Search request can be used to proactively monitor one or more LDAP servers for interesting changes that in turn cause specific actions to be taken by an application. For example, an electronic mail repository may want to perform a "create mailbox" task when a new person entry is added to an LDAP directory, and a "delete mailbox" task when a person entry is deleted from an LDAP directory.

Synchronization

Some LDAP clients, such as those that execute on a portable computer, may maintain a partial or a complete offline copy of the entries stored in an LDAP server. While connected to the network, such a client can direct all queries to the copy of data it holds and use a persistent search to actively maintain the contents of the offline copy. (Alternately, the client could direct requests to the LDAP server that is the source of data.)

Configuring Persistent Search in eDirectory

Before we show you how to use the Persistent Search feature in eDirectory, let's make sure your server is set up properly so you won't run into any brick walls when you fire up your application. These configuration settings apply to eDirectory 8.6.1 (later versions may change slightly).

The configuration information for Persistent Search is stored on the LDAP Server object in eDirectory. Like any other LDAP server configuration attribute, these attributes are manageable through ConsoleOne or through the LDAP Modify operation (you can use an LDIF file with ICE, or the ldapmodify utility).

ConsoleOne version 1.3.2, which ships with eDirectory 8.6.1, has a Persistent Search tab on the LDAP Server object for configuring these attributes. The attributes are as follows:

  • ldapEnablePSearch. This Boolean attribute specifies whether or not the Persistent Search feature is enabled on the LDAP server. The default eDirectory installation sets this value to True, which enables Persistent Search on the server. An administrator who wants to disallow Persistent Search sets it to False. This is provided as a checkbox on the Persistent Search tab on the LDAP Server object in ConsoleOne.

  • ldapMaximumPSearchOperations. This integer attribute is used to specify the number of Persistent Search connections possible on the server. A value of zero means that an unlimited number of Persistent Search operations are allowed. The default eDirectory installation sets this value to 0, enabling an unlimited number of Persistent Search operations. As an administrator, you are advised to set this value to whatever number of connections you deem fit. In a production environment, the recommended value for this is 5 - 10.

    To better grasp the implications of this setting, let's explain more about how Persistent Search works in eDirectory. The first (searching) phase of a Persistent Search operation works like any other search operation, except that the server sets up a little bit of data that it needs to remember for the next phase. The second (events) phase of the operation watches for updates to happen in the directory. Whenever an update happens to an entry, the server effectively re-issues a search operation for that entry on the behalf of the client that initiated the Persistent Search. If the entry matches the original search criteria, and if the client has rights to see the entry, it is returned to the client.

    So, if there are currently 200 active Persistent Search operations and an update happens to an entry, it is (potentially) as if 200 search operations fired off at the exact same time (we're generalizing a bit here, but you get the picture). If your server can handle 200 simultaneous search operations (that is, if it's fast, has a good size cache, and so on), go ahead and set it that high.

  • ldapIgnorePsearchLimitsForEvents. During the "searching" phase of a Persistent Search operation, the normal size and time limitations are considered. The size and time limits are configured in attributes called searchSizeLimit and searchTimeLimit on the LDAP Server object. They may also be passed in as part of the search request.

    The ldapIgnorePsearchLimitsForEvents attribute is a Boolean value that indicates whether the size and time limits configured for the normal search operation should be ignored during the "events" phase of the Persistent Search operation. If this is set to False, the entire Persistent Search operation is subject to the search limits. If either limit is reached, the search will fail with a sizeLimitExceeded or timeLimitExceeded result code. This setting is provided as a checkbox on the Persistent Search tab on the LDAP Server object in ConsoleOne.

    This may seem like an unneeded configuration setting. We agree! We only added it in anticipation that someone out there would want eDirectory to behave the same way one of our competing directory servers does.

Writing Persistent Search LDAP Applications

The Novell LDAP C and JAVA SDKs included in the NDK (Novell Developer Kit) support the persistent search control on the LDAP client side. The NDK is available at http://developer.novell.com/ndk/.

With the C SDK, you first define the Persistent Search criteria using the ldap_create_persistentsearch_control function. This function produces an LDAPControl data type that is then passed to ldap_search_ext or ldap_search_ext_s. The ldap_create_persistentsearch_control function takes as arguments the following:

  • changeTypes that the client is interested in (add/modify/rename/delete)

  • changesOnly indicating if the initial result set needs to be sent

  • returnEntryChangeCtrl specifies whether or not you want to be told why an entry is being returned

  • isCritical will cause the server to return the unavailableCriticalExtension error if it does not support the control; otherwise, unsupported controls are ignored

  • Ctrlp is an out parameter that is then used to initialize the controls array that is passed to the ldap_search_ext call

If the returnEntryChangeCtrl parameter is non-zero, calling ldap_parse_entry_ controls allows you to have the controls that the LDAP server sent back as a part of the entry. The ldap_parse_entrychange_control lets you obtain the changetype of the entry (add/modify/rename/delete) and the previous DN if the entry had undergone a rename operation.

With the Java SDK, you first create a LDAPPersistSearchControl object, specifying the same data as you would in the C function, and then add it to the constraints that are passed to the LDAP connection's search method.

If you'd like to see some sample code using Persistent Search, the Novell LDAP SDKs provide this.

  • The C code samples are in searchPersist.c under the Samples directory of the SDK. They can be viewed at http://developer.novell.com/ndk/doc/ samplecode/cldap_sample/cldap_sample/searchPersist.c.html.

  • The Java example is in SearchPersist.java and can be viewed at http://developer.novell.com/ndk/doc/samplecode/jldap_sample/controls/SearchPersist.java.html.

  • You'll also find sample code for JNDI at http://developer.novell.com/ndk/doc/ samplecode/extjndi_sample/extjndi_sample/PSearch.java.html. This code will give you a head start in configuring your application to use Persistent Search.

Sample Output

Now let's look at what kind of output you can expect when you compile and run the sample program from the C SDK against Novell eDirectory. Assume a sample LDIF file such as the following:

dn:cn=john,ou=finance,o=myorg
changetype:add
cn:john
givenName: john
sn: Lamb
objectclass:inetOrgPerson

dn:cn=bob,o=eng,o=myorg
changetype:modify
add: mail
mail:bob@myorg.com

dn:cn=alice,ou=hr,o=myorg
changetype:moddn
newrdn:cn=mary
deleteoldrdn:1

dn:cn=jack,ou=admin,o=myorg
changetype:delete

You might run the Persistent Search program with a search base of o=myorg, scope as subtree, and search filter as objectclass=inetorgperson. The program will establish the Persistent Search connection to the server, return the initial result set (if you have set changesOnly to False) and wait for updates from the server. Now you can affect the changes in the above LDIF file (assuming you have Bob, Alice, and Jack in the right container in your tree) by using ldapmodify or ICE on your server.

When these changes take place in the eDirectory database, the Persistent Search client is notified of the updates. The output you will see on your screen will be as follows:

dn:cn=john,ou=finance,o=myorg
changeType: Entry Added

dn: cn=bob,ou=eng,o=myorg
changeType: Entry Modified


dn:cn=mary,ou=hr,o=myorg
changeType: Entry Renamed
previous DN: cn=alice,ou=hr,o=myorg

cn:cn=jack,ou=admin,o=novell
changeType: Entry deleted

Note that in the case of a rename, the server sends a previous DN of the entry as a part of the entry change control which is being printed to the screen by the Persistent Search client.

Things to Remember

This section presents some tidbits of information about Persistent Search in general and the eDirectory support for Persistent Search in particular.

  • The RootDSE (DSA-specific entry) in each LDAP server lists, among other information, the controls supported by the LDAP server. As the Persistent Search control is being supported by eDirectory 8.6.1, the RootDSE entry now contains the OIDs of the Persistent Search control on the supportedControls attribute. These OIDs are 2.16.840.1.113730.3.4.3 (request oid) and 2.16.840.1.112730.3.4.7 (response oid). In an application, you may want to examine this attribute to gain knowledge of the server's capabilities. To read the supportedControls in the rootDSE, perform a search where the base is empty (""), the scope is base-level, the filter is (objectClass=*), and the return attributes set to "supportedControls".

  • In the case of an entry being modified, the Persistent Search control does not send any information about the attribute or value that was added or deleted from the entry. This support simply does not exist in Persistent Search. A separate Internet Draft called "LCUP: LDAP Client Update Protocol" has been proposed to the IETF to cover this and other shortcomings in the Persistent Search control.

  • Being a distributed and replicated directory, eDirectory can have its data partitioned across several servers in the same eDirectory tree. The Persistent Search support in eDirectory 8.6.1 is restricted to the changes that happen in the local server's DIB (directory information base) only. If a change happens in remote server's DIB and the same information is not hosted in the local sever, the client will not be notified of the change-even if the change qualifies the Persistent Search client's search criteria. This limitation is because the Persistent Search support in eDirectory is built around the events that happen in the local server only. However, modifications that happen in a remote server and are then synchronized to the local server through the eDirectory replica synchronization mechanism will be sent back to the Persistent Search client as if they happened locally. For this to work, the local server must hold the partition of the data that changed in the remote server.

  • Regardless of the referral settings on the LDAP Server object, referrals are always generated when a part of the search subtree is not held on the local server. This allows the client application to follow those referrals and initiate Persistent Search operations on other servers in order to maintain a full view of the data it is interested in.

  • The Persistent Search operation is memory- and connection-intensive for the LDAP server, as it not only needs to maintain an open TCP connection but also needs to maintain the search request associated with that connection. Every change that happens in the directory database needs to be evaluated against the search criteria of the Persistent Search client, and the client is notified of the change or not accordingly.

    It has been observed that beyond 50 concurrent Persistent Search clients, the performance of eDirectory may suffer. Hence, it is recommended that the maximum concurrent LDAP Persistent Search connections be restricted to the 5 - 10 range via the ldapMaxPSearchConnections attribute on the LDAP Server object. Of course, these numbers are all subject to differences in server capabilities. A multi-processor machine with lots of memory will be able to service many more concurrent Persistent Search operations before any performance degradation is noticed.

  • The error codes that can be returned during a Persistent Search operation are all similar to the normal LDAP search operation. Some of the common error codes you may encounter are "sizeLimitExceeded" or "timeLimitExceeded" if ldapIgnorePsearchLimitsForevents is set to False and the searchSizeLimit or searchTimeLimit has been set to some value other than 0 (0 means unlimited). A referral error could be returned with an ldapreferral if the search spans beyond the partitions that are hosted on the local server. You could also encounter the adminLimitExceeded error if the maximum Persistent Search connections limit is exceeded.

  • The Persistent Search operation is incapable of de-referencing alias entries in the directory. So if the Persistent Search request specifies de-referencing while searching entries or finding the base object, such a request will not be honored by the LDAP server and an error code of 80 (LDAP other) will be sent back to the client.

  • eDirectory does not yet support Persistent Search on the subSchemaSubEntry (cn =schema) and RootDSE. That means any change in the schema of the directory or in the RootDSE information is not communicated back to the client that is doing such a search.

Conclusion

Hopefully this AppNote has given you a working understanding of what Persistent Search is and how to build your applications using the LDAP C SDK and Java SDK support for it. We hope you find this new feature in eDirectory useful as you write more directory-enabled applications and unleash the power of the LDAP directory.

* Originally published in Novell AppNotes

Disclaimer

The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.

© Copyright Micro Focus or one of its affiliates