Database Manager for NetWare
Articles and Tips: article
01 Jan 1998
The Database Manager for NetWare allows NetWare 4.x servers to function as a middle-tier platform for remote database connectivity, enabling access to the leading database engines regardless of location or type of information being requested. This article discusses such topics as the heterogeneous multi-join engine, ODBX architecture, and data query processing.
Introduction
An explosion of interest in the Internet and intranets, leveraged by the ability of Web developers to dynamically link Web page content to today's powerful database engines, has created a significant requirement for database connectivity on the NetWare platform.
Presently, the World Wide Web consists of well over 120,000 connected servers supporting millions of users. As the Web doubles in size approximately every 53 days, it is constantly consolidating Web content into a colossal virtual disk loaded with every imaginable form of information. As this cyber-disk continues to grow and expand exponentially, so does the requirement for innovative and powerful database connectivity technologies that can extract everything from flat- file documents to "sound bytes" to full-length movies.
As a result of this technological explosion, Novell has made significant commitments to today's Internet technologies. Novell is investing heavily in the area of Java technology as it aggressively develops innovative middle-tier solutions which will carry it forward into the next century. One of these new technologies is the Database Manager for NetWare. This technology allows NetWare 4X servers to function as a middle-tier platform for remote database connectivity, enabling access to the leading database engines regardless of location or the type of information being requested in the query.
As this data is harnessed and consumed by the Web development community, much of it is taking the form of binary large objects (BLOBs).Since the Web interface encourages the distribution of files, graphics, and other large binary objects, basic file handling is as important as ever. Due to the widespread availability of Web browsers, servers must be scalable enough to quickly enable expanded usage (both internal and external).
Novell is determined to meet the challenge of providing a database connectivity framework for instant access to any type of information which resides in a database, even when it is to be accessed through multiple tiers and on disparate platforms. The Database Manager for NetWare performs the function of a data request broker by providing a multi-tier framework connecting to multiple platforms and multiple databases throughout the enterprise using any combination of JDBC, ODBC, Basic scripting or OLE DB-compliant applications.
Heterogeneous Multi-Join Engine
Unlike the majority of conventional database drivers, which access a single physical data source, Database Manager for NetWare utilizes powerful multi-join technologies which can access and join multiple data sources in a single database query.
From a client viewpoint, this allows multiple distributed databases to appear as one logical database to the NetWare 4.X server. The Database Manager includes client, data broker and data server components. These components have the capability of residing on either the same or disparate computer systems. The data server components can support end-user applications authored using ODBC, JDBC or OLE DB tools. Client-to-server or server-to- server implementations are supported utilizing these developer interfaces.
Database Manager for NetWare implements an extended form of ODBC-like connectivity, which we refer to as the ODBX engine (Open Database Exchange). One of the unique capabilities of this technology is its support for DBA utilities which maintain a global data dictionary--providing direct, interactive access to all database tables requiring admission within the enterprise. The data broker components support client connectivity over a network, including query, parsing, optimization and distribution within heterogeneous databases.
The data server components support connectivity to physical databases or flat files and will be used when a database vendor's connectivity software is not available. Components of Database Manager for NetWare are listed below:
|
ComponentName
|
Functionality
|
|
ODBX-ODBC |
Microsoft ODBC API |
|
ODBX-JDBC |
Javasoft Java SQL Classes |
|
ODBX-OLE-DB |
Microsoft OLE Database DataProvider |
|
ODBX-DDMGR |
Database DictionaryManager |
|
ODBX-ISQL |
Interactive SQL |
|
ComponentName
|
Functionality
|
|
ODBX-ORB |
Data Request Broker |
|
ODBX-DD |
DataDictionary |
Data Server
|
ComponentName
|
Functionality
|
|
ODBX-ORACLE |
Oracle Database Connection |
|
ODBX-MS-SQL |
Microsoft SQL Server DatabaseConnection |
|
ODBX-SYBASE |
Sybase Database Connection |
|
ODBX-INFORMIX |
Informix DatabaseConnection |
|
ODBX-FILES |
Flat File DatabaseConnection |
|
ODBX-INGRES |
Ingres Database Connection |
|
ODBX-REMOTE |
Remote DatabaseConnection |
ODBX Architecture
As mentioned previously and outlined in the table above, ODBX architecture includes the client, data broker and data server components. The architecture was designed to provide maximum flexibility for implementing enterprise-wide development and deployment of database connectivity configurations. The components are modularized so they can be installed on multiple tiers within any network which has implemented at least one NetWare 4.X server. The following example illustrates one of many possible configurations.
Three Tier Application (Internet Web Server)
A three-tier application example is a Java applet running on a Web server and invoked from the thin client webWeb browser on a Windows desktop.
Database Manager Diagramclient Components ODBX-ODBC
ODBC (Open Database Connectivity) is an API (Application Programming Interface) developed by Microsoft. It is Microsoft's implementation of the X/Open SQL CLI (Call Level Interface) that defines how client/server interactions are implemented for database applications. ODBC also supports the SQL grammar and syntax specified in the ANSI SQL-92 standard. ODBC was developed for the purpose of providing vendor- neutral access to data sources from a client application. With ODBC, a client application is not restricted to any one proprietary interface, and the client application can connect to any ODBC-compliant data source.
ODBX-ODBC is a Microsoft-compliant implementation of the ODBC API (version 2.5). As such, any client application or application development environment can connect to ODBX- ODBC. Once an application is connected to ODBX-ODBC, it can access database tables from multiple heterogeneous networked databases and can join heterogeneous database tables in a single query.
ODBX-JDBC
JDBC (Java Database Connectivity) is a vendor- neutral API developed by JavaSoft. JDBC is a standard part of Java and included in JDK (Java Development Kit) 1.1. The interface is very similar in concept to Microsoft's ODBC API and provides Java programmers with a uniform database interface to a wide range of relational databases. Like ODBC, a JDBC application is not restricted to any one vendor's interface, and the Java application can connect to any JDBC- compliant data source.
ODBX-JDBC is a 100 percent Java JDBC- compliant implementation of the JDBC API (version 1.1). Any Java application, applet or application development environment can connect to ODBX-JDBC. Once an application is connected to ODBX-JDBC, it can access database tables from multiple heterogeneous databases and can join heterogeneous database tables in a single query.
ODBX-OLE-DB
The Microsoft ODBC API is designed to provide access to SQL relational data. While the data can originate from more than one data source, the data accessed by ODBC is assumed to follow the relational data model.
In contrast, OLE-DB, also from Microsoft, is designed to provide access to all types of data. OLE-DB is a set of OLE (Object Linking and Embedding) interfaces that provide applications with uniform access to data stored in diverse information sources, regardless of location or type. These interfaces allow data sources to share their data through common mechanisms.
OLE-DB is built on Microsoft's DCOM (Distributed Common Object Model), which allows access to data sources throughout an enterprise and includes the SQL functionality defined in ODBC. For example, an application might require access to database tables, E-mail, voice dictation, images, and directory services, where data is divided between local and remote systems.
Active Data Objects (ADO) is a set of programming objects from Microsoft that provides a simpler interface for data access than the OLE-DB interface. The ADO objects (Connection, Command, Parameters, Record set, Field and Property) are invoked from ADO-compliant applications such as Visual Basic or Visual C++ and call the OLE-DB interface as required. ODBX-OLE-DB is an OLE-DB-compliant data provider interface. A data provider has a basic requirement to expose data in tabular form, regardless of the type and source of the data, using a predefined set of objects and methods. These objects and methods are used in applications which can draw from several sources of data.
ODBX-OLE-DB allows heterogeneous database tables to be accessed in an OLE-DB or ADO application, along with other diverse sources of data.
ODBX-DDMGR (Data Dictionary Manager)
ODBX supports access to multiple physical databases by maintaining a data dictionary. The data dictionary resides on the same node as the ODBX data broker components, which can be the same or a different node than other ODBX components. This data dictionary has a metadata catalog with entries which point to physical database metadata.
A client application built using ODBX will be independent of the physical database and can join heterogeneous database tables. The application is only aware of the data dictionary metadata information (table names, column names, etc.) and not the physical source of the data. The physical database source can be changed, for example, from the local files to Oracle, or from Oracle to MS-SQL, with no change required in the application.
ODBX-DDMGR is an ODBX client application for creating and maintaining ODBX data dictionaries, for importing physical metadata information and exporting SQL definitions. ODBX-DDMGR can also create and maintain local or remote flat files (sequential or indexed) for use in database applications where a physical database is not required or not available. Flat files can also be used as readily available data source for an application under development, especially if the metadata structure is also under development.
ODBX-ISQL (Interactive SQL)
Structured Query Language (SQL) is a well defined ANSI (American National Standards Institute) standard language for database access. Database vendors strive to support grammar and syntax of the SQL language, but inevitably change implementations to support database-specific features.
ODBX-ISQL is an ODBX client application whose language is based on the ANSI SQL-92 standard. ODBX-ISQL implements a database-independent query language that eliminates the need to learn proprietary vendor dialects of SQL, not all of which are SQL-92 compatible. ODBX-ISQL also provides enhanced data type support that may be missing from a vendor-specific database. ODBX-ISQL is useful for interactive query and update of physical database tables and local files defined in the data dictionary and can be used to test SQL syntax before embedding it in an application. ODBX-ISQL can also be used to test database connectivity independent of a client application.
Data Broker Components ODBX-DD (Data Dictionary)
The ODBX data dictionary resides on the same node as the ODBX data broker components. This data dictionary has a metadata catalog; entries in the catalog point to the physical database tables or database views. Database views are very useful when the application only needs a subset of fields in a table or the application requires frequent joins of multiple database tables. The data dictionary can also have multiple entries (synonyms) for the same database table or view. Synonyms can be very useful in database applications that require self-joins.
Metadata information stored in the data dictionary includes the following information relative to each physical database table.
Table Name
Column Name, Data Type, Nullability
Primary And Foreign Keys
Indexes
Database Type (Oracle, Ms-sql, Etc.)
Database Name And Location
This metadata can be imported from a physical database to the data dictionary. The data dictionary also supports sequential or indexed local files as a source of relational data.
The ODBX data dictionary supports client applications which are truly database-independent. The physical database can be changed without any effect on the application. In contrast, even though ODBC provides a vendor-neutral programming interface, ODBC itself is not database-independent. In particular, some database vendors do not support ANSI SQL data types for time stamp and time intervals.
Consequently, ODBC applications must be aware of which data types are supported. By contrast, ODBX is truly database-independent. If an ANSI SQL data type is not supported by the physical database, the data type is emulated by the ODBX data dictionary and mapped to an appropriate physical database type.
ODBX-DRB (Data Request Broker)
ODBX contains a data request broker which is a sophisticated SQL processor, optimizer and distributor. The SQL processor supports full read/write capability for all database connections. SQL processing begins by decomposing an SQL statement (which may have embedded subqueries) into its constituent parts:
Data Query Statements (Read Data) Select
Data Manipulation Statements (Write Data) Update
Insert
Delete
Data queries are processed first, since the results may be needed for a data manipulation statement. A specific example might be as follows:
update emp set birthday =(select birthday from personnel where empno = 1234)
This example would require the select statement to be processed first, since the result is needed for the update statement.
Data Query Processing
ODBX-DRB can decompose and distribute data queries as close to the data source as possible. ODBX-DRB has the ability to decompose a query that includes tables from multiple databases into subqueries for each database, which are then executed within each database for maximum performance and minimum network traffic.
Decomposition of a query also leads to decisions regarding how much of the query can be pushed down into the physical database. Examples of pushing-down operations are join clauses, including where clauses and order clauses. The more homogenous the query (i.e., strictly involving one database), the higher the number of clauses which can be executed in the physical database. ODBX-DRB will determine how much of the query can be sent directly to the database and how much must be processed by ODBX-DRB, pushing as much processing as possible down to the underlying database as a means of achieving the best possible performance.
ODBX-DRB has the capability to rewrite a database query based on keys and indexes in the physical database. ODBX-DRB also distributes a heterogeneous query involving more than one database through the use of cardinality and selectivity of the tables involved in the query. This information determines the order and sequence of interactions with the various databases. The primary goal of query optimization is to reduce the bandwidth requirements between ODBX and the underlying physical databases.
In addition, ODBX-DRB will rewrite the query as necessary to handle database-specific SQL dialect issues. For example, conversion of data from string (e.g., used in a qualifier) to an Oracle date will use the Oracle TO_DATE function, and conversion to an MS-SQL date will use the MS-SQL CONVERT function.
Data Manipulation Processing
After the data query processing finishes, database change processing starts, perhaps using intermediate results from the query processing. For these commands, the SQL statement is also decomposed, processed and forwarded to the physical database. The processing may include rewriting the statement for specific dialects.
Data Server Components
The ODBX Data Request Broker connects to the physical databases using the ODBX Data Server components. These data server components are ODBX database drivers for each supported database (or for flat files). The database drivers then connect to the physical databases in one of three ways:
ODBX Direct Connection to Vendor-Supplied Connectivity Library
ODBX Network Connection to Vendor-Supplied Connectivity Library
ODBX Network Connection to A Remote ODBX Data Request Broker
ODBX Direct Connection To Vendor Library
The most direct connection form is to implement the vendor-supplied connectivity library running on the same computer system as the ODBX data broker components. The vendor library will then connect to the vendor database which is executing on the same machine as the database server or running remotely to the machine serving up the vendor's database stores. A vendor library will only be available for a direct connection to the physical database if it runs on the same computer system as the ODBX-DRB. Not all vendor libraries are available on all types of platforms.
The following table shows the ODBX database drivers and the respective vendor libraries:
Data Broker
|
DatabaseDriver
|
VendorLibrary
|
|
ODBX - ORACLE |
Oracle Database Connection OCI (Oracle CallInterface) |
|
ODBX - SYBASE |
Sybase Database ConnectionDBLIB |
|
ODBX -MSSQL |
MS-SQL Server Connection DBLIB(Windows Only) |
|
ODBX - DB2 |
IBM DB2 Database ConnectionDDCS |
|
ODBX - INFORMIX |
Informix ConnectionNETLIB |
|
ODBX - INGRES |
Ingres DatabaseConnectionINET |
|
ODBX - RDB |
Oracle/RDB Connection DSRI (Open VMSOnly) |
|
ODBX - FILES |
Flat File DatabaseConnection |
A vendor library will only be available for a direct connection to the physical database if it runs on the same computer system as the ODBX-DRB. Not all vendor libraries are available on all types of platforms.
ODBX Network Connection to Vendor Library
If a vendor-supplied connectivity library is not available on the same node as the ODBX-DRB, then an ODBX database driver can connect across the network to the vendor-supplied connectivity library running on another system.
ODBX Network Connection To Remote ODBX-DRB
In large enterprise systems, there might be a requirement to interconnect the ODBX Data Request Brokers. This requirement can be met using ODBX-REMOTE. ODBX-REMOTE allows ODBX-DRBs to be interconnected.
This interconnection capability allows access to enterprise data which might be organized into several data dictionaries.
Usage
The steps required to use ODBX will vary from site to site, and within systems at a given site. These steps include installation, configuration, testing and programming. Installation and configuration are DBA (Database Administrator) activities, and programming is an application development activity.
Installation
ODBX installation must be planned around which computer systems are client systems, which systems are application or Web servers, and which systems are being used as database servers. Use of ODBX will typically require third-party software, such as ODBC driver managers, Java development kits, database vendor connectivity software and database server software.
Configuration and Testing
ODBX configuration and testing uses the client components ODBX-DDMGR and ODBX-ISQL. These components run on a client system and access data dictionaries on an application or Web server. Data dictionaries can then access data from physical databases. ODBX-DDMGR is used to create the ODBX data dictionaries and to import metadata information from physical databases on database servers. ODBX-DDMGR can also be used to create flat file databases for use in database and application prototypes.
ODBX-ISQL is used to query the data dictionary tables (and hence the physical database tables) using ANSI standard SQL statements. Thus, the data dictionary can be tested before a client application is available. SQL queries which might be used in the application can be tested for syntax, performance and efficiency. Database traces can be used to look for performance bottlenecks (such as joins using non-unique indexes or sequential searched through large tables).
ODBX-ISQL can also be used for data migration and replication, even across heterogeneous databases. Since physical database tables from heterogeneous databases can be catalogued in a single data dictionary, relatively simple SQL statements (such as insert into mssql_table select * from oracle table) can be used to migrate data from one database to another (assuming the source and destination tables have similar columns and data types). The use of ANSI- standard SQL data types allow migration (even joins) of dissimilar physical data types, such as Oracle and MS-SQL date stamps.
Programming
Once the data dictionary and data access has been tested, application development can begin. Application development typically uses one of the following scenarios:
1. Application development tools which support one or more of the following interfaces, ODBC, JDBC, or OLE-DB.
2. Programming languages which can call the ODBC API, Java SQL classes or OLE objects.
Regardless of the choice of application development environments, the use of ODBX is transparent because of adherence to ODBC, JDBC and OLE-DB standards.
Conclusion
For the enterprise that requires transparent simultaneous access to disparate database schemas hosted in a cross-platform environment, Database Manager for NetWare can provide access to a broad range of heterogeneous physical databases.
It offers a more sound, logical, functional implementation of ODBC, with a robust multi-tier framework for connecting to platforms and databases throughout the enterprise using any combination of JDBC, ODBC, Basic scripting or OLE DB-compliant applications. Database Manager for NetWare will also provide scalability that is relative to that of the NetWare platform. In fact, early performance measurements show that throughput gain on a SQL query is substantially greater on the NetWare platform than either Windows or Unix environments.
As Internet/intranet information continues growing and expanding, so will the requirement for innovative and powerful database connectivity that quickly extracts everything from flat-file documents to full-length movies. Where ODBC is the primary developer implementation now, JDBC will be increasingly important for moving enterprise-level information access to the future.
As the Java platform matures, this, along with the multiprocessing kernel of Moab, Novell's next- generation server operating system, will deliver multiple instantiations of the JVM for peak network loads, increasing performance by an order of magnitude. In particular, the superior I/O, reliability, and optimized resource-sharing abilities of the NetWare platform will make it ideal for enterprise-level data access. Novell's first version of the Database Manager for NetWare product is scheduled for release early next year.
* Originally published in Novell AppNotes
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.