=head1 NAME PXB: Perl XML data Binding =head1 AUTHOR Maxim Grigoriev =over 1 =item L</"INTRODUCTION"> =item L</"PXB FRAMEWORK"> =item L</"DATA MODEL"> =item L</"API CLASS INTROSPECTION"> =item L</"CASE STUDY"> =over 2 =item L</"Utilizing PXB for perfSONAR-PS web services"> =item L</"perfSONAR-PS data model for PingER service"> =item L</"SQL MAPPING"> =back =item L</"REFERENCES"> =back =head1 INTRODUCTION This document presents and describes API designed for binding RelaxNG (Compact) or XML schema, represented by RelaxNG Compact notation into the perl class library. The main idea behind such binding is to provide flexible and scalable API which will allow developers of the document literal based web services easily describe any service messaging profile by writing simple perl data structures. This approach allows you to work with data from any XML message using your own class structures, so your XML elements are binded into the perl objects, therefore it called Perl XML data Binding (or PXB). The PXB framework will handle all the details of converting data to and from XML or DOM objects based on your instructions. The DOM objects tree is created by utilizing very fast LibXML library. PXB was designed to allow you a high degree of control over the translation process with custom callbacks and SQL mapping of the hierarchical XML entities into the flat SQL database schema. Even more importantly to note that PXB creates heavily documented, highly supportable and maintainable perl class hierarchies. It also automatically creates test suit for every created class. The L<Perl::Tidy> and L<Perl::Critic> profile files are included into the standard test suit in order to comply with the Best Perl Practices. If you familiar with java XML data binding packages such L<xmlbeans|/"2. xmlbeans">, L<CASTOR|/"4. CASTOR">, L<METRO|/"3. METRO"> or older L<JAXB|/"5. JAXB"> then idea of unmarshalling XML into the simple objects should be known to you. =head1 PXB FRAMEWORK =head2 or Why Yet Another XML framework PXB uses the power of perl data structures to define the rules of how perl objects are converted to or from XML (the binding). Or more explicitly to and from the DOM tree objects. Why not to use DOM instead? DOM is just a representation of the XML tree data structure. It does not provide any facility to hook up easily some custom callback aligned with some external data schema to the particular element in the tree. The current spectrum of available XML APIs for perl is limited to tree walking and XPath based search. Also, the process of refactoring heavy XML driven perl code based on XML/XPath approach was found to be very tedious and error prone. Especially when schemas are constantly adapting to the ever changing customer's requirements. The PXB framework is an attempt to bring configurable callbacks into the DOM and add some functional validation inside of such binding as well as provide mechanism to map SQL database schema to the hierarchical structure of the DOM tree. Importance of such mapping technique was exposed by the lack of robust support for native XML data type among the current list of “freeware” storage engines where the SQL RDBM is more or less a standard and performance of the native XML databases are still far from the optimal. Also, PXB uses perl as meta-language and it uses perl to create perl OO API which makes schema's code significantly smaller. Any developer can concentrate on implementing semantic rules, protocols and actual functionality instead of wasting any time on reimplementing once again XML tree walking API. Moreover, there is an automated test suite for the each class of the generated API and every method call is documented. The internal structure of every class was designed to be clean, readable and easily understandable by any other perl developer. It utilizes L<perl’ best practices|/"6. Perl Best Practices, by Damian Conway"> and uses C<fields> packages to provide tighter encapsulation and creates explicitly named lists of accessors and mutators. Behavioral semantic of the each class in the class tree is the same and implements absolutely the same methods to allow transparency and propagation of the implemented callbacks across the API. =head2 DATA MODEL The basic XML element definition is represented as perl hash reference: =over <element-variable> = '{ attrs => { ' <attributes-definition> ',' ‘ 'xmlns’ ‘ => ‘ ' <namespace prefix> '},' 'elements => [' <elements-definition> '],' '‘text ’ => ' <text-content> ',' '‘sql => {' <sql-mapping-definition> '}' '}' =back Where each <xxxx-definition> can be expressed in L<EBNF|/"7. EBNF> as: =over <attributes-definition> = (attribute-name '‘=>’' attribute-value) (',’' <attributes-definition>)* attribute-name = string namespace-id = string attribute-value = ‘scalar’| ('’enum:’'(attribute-name ( '‘,’' attribute-name)*)) <elements-definition> = ( '‘[‘' element-name '’=>'’ ( <element-variable> | ’'[' ‘ <element-variable> '’]’' | ’'[‘' ( <element-variable> '’,'’)+’ ']’' | '’[‘' ('['‘ <element-variable> '’]’,'’?)+ ’']'’ ) ',' conditional-statement? ']’')* conditional-statement = ('‘unless’'|’'if'’)’':'’(registered-name ('‘,'’ registered-name)*) registered-name = attribute-name|element-name element-name = string <text-content> = ‘scalar’|conditional-statement <sql-mapping-definition> = (sql-table-name ‘'=>’ ‘{'‘ sql-table-entry '‘}’' ) ( '‘,'’ <sql-mapping-definition>)* sql-table-entry = (sql-entry-name '‘=>’ ’{'‘ entry-mapping ‘'}’' ) ( '‘,’ ' sql-table-entry )* sql-table-name = string sql-entry-name = string entry-mapping = '‘value’ ‘=>' ’( element-name|( '‘['‘ element-name ('‘,'’ element-name)+ ’']’')) ( ‘','’ if-condition)? if-condition = '‘if’ ‘=>’( ' attribute-name '‘:’' attribute-value | '[' ( ' attribute-name ':' attribute-value ',' )+ ']' ) =back Where: =over =item elements => [parameter => [$parameter]] - defines list of 'parameter' sub-elements =item elements => [parameter => $parameter] - defines single 'parameter'’ sub-element =item elements => [parameters => [$parameters,$select_parameters]] - defines the choice between two single ‘'parameters' sub-elements of different type =item elements => [datum => [[$pinger_datum], [$result_datum]] ] - defines choice between two lists of 'datum' sub-elements of different type =back In C<elements-definition> the third member is an optional conditional statement which represents validation rule. For example I<‘unless:value>’ conditional statement will be translated into the perl’ conditional statement C<< && !($self->get_value) >> where ‘value’ must be registered attribute or sub-element name and this condition will be placed in every piece of code where perl object is serialized into the XML DOM object or unmarshalled from it. =head2 API CLASS INTROSPECTION In order to provide the same class interface throughout the API, the same universal constructors and same set of method calls were implemented. Currently the constructor body includes initialization part as well and this part is slightly different for different modules, but it might be reduced to generic constructor and some custom initialization part in the future where generic constructor might be inherited from some sort of the base Object class. Currently every constructor “knows” how to initialize the whole class from the DOM object, XML string fragment or from the reference to the perl hash structure of the named class fields. Also, it “knows” how to handle arrays of the class fields with support for elements identified by I<id>. It supports special field, named idmap for that purpose. It “knows” how to serialize object of the class into the DOM or XML text string and how to map contents of the object on some SQL schema. By issuing I<registerNamespaces> call to the root object one can obtain the reference to the hash with all namespaces utilized by every sub-element in the object. It utilizes another special field, called C<nsmap> for that. This field is an object of the generated helper class and serves as container for map between local names and namespace prefixes. Every namespace is identified by namespace prefix and the version of the schema is getting built into the generated API package pathname. =head1 CASE STUDY Our case is based on building interoperable SOAP document/literal based webservice for L<perfSONAR-PS|/"8. perfSONAR-PS"> project. Webservices are indeed wrappers around network performance monitoring tools. Where there are two types of services - Measuremnt Archive (MA) with ability to publish historical data and Measurement Point (MP) providing on-demand measurements. =head2 Utilizing PXB for perfSONAR-PS web services The major utilization scenario of the PXB framework comes when there is a need to build MA service with SQL database based storage engine. In any case PXB framework will provide the complete solution and actual development for MP will be contained in writing real time measurements facility class and then utilizing PXB to inject measured data into the published XML message. For MA the whole SD process will be dedicated to writing the actual message handler and implementing actual protocol specifications. For example, for PingER MA it is L<perfSONAR_PS::Datatypes::PingER> class, inherited from L<perfSONAR_PS::Datatypes::Message> class with actual implementation of SetupData request and MetaDataKey request handlers. The L<perfSONAR_PS::Datatypes::Message> is an abstract class, extending L<perfSONAR_PS::Datatypes::v2_0::nmwg::Message> class. This extra class exists because the root B<Message> class from the PXB is message type agnostic. =head2 perfSONAR-PS data model for PingER service The root of the perfSONAR-PS schema and the root of the OO API built by PXB is the B<Message> object. It exists in the perfSONAR base namespace identified by C<nmwg> id. The schema is versioned. The most current root package name for B<Message> class is L<perfSONAR_PS::Datatypes::v2_0::nmwg::Message>. The base schema is completely defined by L<perfSONAR_PS::DataModels::DataModel> module. This is a simple perl package and not the class, because it has only data definitions. The current B<DataModel> has implemented definitions from several B<OGF> schemas, namely: C<filter.rnc, nmtopo.rnc, nmbase.rnc, nmtm.rnc, nmtl3.rnc, nmtl4.rnc>. There is no SQL mapping definitions in the base data model allowed, because SQL mapping is a service specific. Any service specific extension of this base schema must be extended in the separate data model package as it was done for PingER service. The PingER data model can be viewed as an example and it is contained in the L<perfSONAR_PS::DataModels::PingER_Model>. Any other extension data model package can be created for any other service. Let’s look on example of the parameter element. It is described in the base model as: =over $parameter = {attrs => {name => 'scalar', value => 'scalar', xmlns => 'nmwg'}, elements => [], text => 'unless:value', }; =back That means the parameter element has name and value attributes, does not have any sub-elements and might have optional (only when value attribute is undefined) text content. For PingER service it was extended as: =over $parameter->{'attrs'} = {name => 'enum:count,packetInterval,packetSize,ttl,valueUnits,startTime,endTime,deadline,transport,setLimit', value => 'scalar', xmlns => 'nmwg'}; =back Where explicit enumeration was added for the list of allowed values of the name attribute and SQL mapping was defined as: =over $parameter->{sql} = {metaData => {count => {value => ['value' , 'text'], if => 'name:count'}, packetInterval=> {value => ['value' , 'text'], if => 'name:packetInterval'}, packetSize=> {value => ['value' , 'text'] , if => 'name:packetSize'}, ttl=> {value => ['value' , 'text'], if => 'name:ttl'}, deadline => {value => ['value' , 'text'] , if => 'name:deadline'}, transport => { value => ['value' , 'text'], if => 'name:transport'}, }, time => { start => {value => ['value' , 'text'], if => 'name:startTime'}, end => { value => ['value' , 'text'], if => 'name:endTime'}, }, limit => { setLimit => { value => ['value' , 'text'], if => 'name:setLimit'}, }, }; =back Please note that C<time> table name was made up to allow mapping of the time range related entities. Also, C<tlimit> table name was made up to provide non-existed paging mechanism in order to limit the size of the requested dataset in the response message. As it clear from the example above, the schema can be modified in any time without affecting the rest of the service API. For example extra attribute ‘named C<type>’ can be added and the whole API framework rebuild in the matter of seconds. Then developer can utilize this extra attribute type by calling C<< ->get_type >> on the C<parameter> object in the message handling class and new functionality will be immediately supported. =head2 SQL MAPPING Lets look again on the example of the SQL mapping definitions for PingER service from previous chapter. Every object of the API class tree implements C<querySQL> call. It accepts reference to the empty hash and passes this reference through the whole objects tree while filling contents of the hash with initialized contents of the “objects of interest”. The “object of interest” is defined by the SQL mapping definition. For example, for definition: =over C<< {metaData => {count => {value => ['value' , 'text'], if => 'name:count'} >> =back for C<Parameter> object with C<name> attribute set to ‘C<count>’ and C<‘value> attribute’ or C<text> content set to ‘100’ it will add: C<< metaData => {‘count’ => { ‘eq’ => ‘100’}} >> into the SQL query hash. The resulted hash for C<metaData> table will be returned and will look as: =over C<< { metaData => {count => '100'}} >> =back where it can be easily passed to any of SQL ORM frameworks in order to build proper C<WHERE> clause. For example L<Class::DBI> or with minor refactoring to the L<Rose::DB> =head1 REFERENCES =over =item 1. RelaxNG Compact L<http://relaxng.org/compact-tutorial-20030326.html> =item 2. xmlbeans L<http://xmlbeans.apache.org> =item 3. METRO L<http://java.sun.com/webservices/index.jsp> =item 4. CASTOR L<http://www.castor.org/> =item 5. JAXB L<http://java.sun.com/developer/technicalArticles/WebServices/jaxb/> =item 6. Perl Best Practices, by Damian Conway =item 7. EBNF L<http://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form> =item 8. perfSONAR-PS L<https://wiki.internet2.edu/confluence/display/PSPS> =back =cut