XCAT-C++: User Guide

1. What is the CCA Component Model?

   The Common Component Architecture (CCA) specification is an initiative to develop a common architecture for building large scale scientific applications. CCA places minimal requirements on components to facilitate the integration of existing scientific libraries into a CCA framework and also to minimize the impact of the component layer on performance. The component specification of CCA is expressed as a set of interfaces that precisely state the expected behavior for component-to-component and component-to-framework interaction. The specification does not mandate the use of any specific form of distributed or parallel technology as the underlying communication architecture, thereby ensuring that it does not preclude applicability to serial, parallel, distributed or Grid systems. This approach explicitly facilitates research groups to focus on utilizing the same high level component specification for a wide range of distributed and parallel applications. The functionality provided by a component may be used in a wide variety of applications. Also, a number of different components can provide the same functionality by implementing the same set of component interfaces.

   • The CCA Forum defines a component as a software unit that interacts with other components encapsulating well-defined functionality, or a set of functionalities. A component can consist of multiple processes (an MPI job, for example), or multiple components could all be running within a single process. Some fundamental CCA concepts include ports, services objects, component identifiers, and the builder service.

   • Communication between CCA components takes place via their ports, which follow a uses/provides design pattern. A provides port is the public interface implemented by a component. It can be referenced and used by other components. It can also be viewed as the set of services that are exported by the component. A uses port is a connection endpoint that represents the set of functions that it needs to call. Port descriptions for CCA components are provided using the SIDL specification. CCA applications are composed by connecting the uses port of one component to the provides port of one another. The CCA specification does not state the mechanism by which calls are transferred from the uses port to the provides port of the connected component. This is considered to be an implementation specific detail that is hidden from the end user and handled seamlessly by the underlying framework.

   ---

2. XCAT-C++ dependencies:
   • g++ version 3.3.5. Currently, the code does not work with g++-4.0.
   • The following libraries from xct are used by xcat-c++: libldmalloc.so, libmsl.so, libproteus.so, libwsit.so, xbmp.so
   • python, version 2.4 or newer
   • swig
   • ssh should be setup on all the target machines in a such a way so that the XCAT-C++ user can ssh to them without typing in a passphrase or password.

   ---

3. Compiling XCAT-C++.
   • Edit the top level makefile.config file:
     • set g++ to point to version 3.3.5

     • You need to use python 2.4 for the scripting front-end of XCAT-C++.

     • If you use the deps/ directory, then you need to just skip to the next section of this user guide.

     • Edit this line to provide the correct path to XERCES
       XERCES_LIB_PATH=$(XCATCPP_HOME)/../deps/lib

     • Edit this line to provide the correct path to PYTHON_HOME (version 2.4)
       PYTHON_LIB_PATH=$(XCATCPP_HOME)/../deps/lib

     • Make sure the path to includes of python is correct PYTHON_INCLUDES=-I$(XCATCPP_HOME)/../deps/python2.4/include

     • Make sure that "swig" (http://www.swig.org) is in your path
     
     

   • Debug or Release version: Set the variable "MODE" in makefile.config to either "dbg" or "rel" for compiling in debug and release mode respectively.
     

   • Edit the source code to specificy the environment of the target machines.

     • Edit the file $(XCATCPP_HOME)/python/run.py to use the python front-end

     • Set the correct values for typemapProvides and typemapUses.

     • typeMapProvides.putString("fullExecPath", sys.path[0] + "/../samples/components/printer/printer")
       This line specifies the location of the executable "printer".

     • typeMapProvides.putString("remoteHost", "rommel.cs.binghamton.edu")
       This line specifies the host on which the component "printer" should be launched.

     • typeMapProvides.putString("creationProto", "/usr/bin/ssh")
       This line specifies the path to ssh on the host mentioned above.

     • typeMapProvides.putString("creationArgs", "-l " + sys.path[0] + "/../libs/dbg/xct/xbmp.so -d 10")
       This line specifies the path to the binary provider for proteus that will be used by the "printer" component for communication. The option "-d 10" specifies the debug level with which the component should be launched.

     • The entries for typeMapUses should be interpreted similarly.
     
     

   • Edit the file $(XCATCPP_HOME)/samples/components/compose/Compose.cpp

     • Either manually set the value of XCATCPP_HOME or set it as part of your shell environment so that it can be picked from the environment.
     
     

   • make clean; make all

   ---

4. Setting a debug level for XCAT-C++ components

   • Convention for setting the debugging levels:

     • Proteus Library: debug statements will be enabled if the level is set to greater than or equal to 50.

     • XCATCPP Library: debug statements will be enabled if the level is set to greater than or equal to 30 and above.

     • Sample Components: debug statements will be enabled if the level is set to 29 or higher.

     ---

   • Obtaining name of the machine along with each line in the debug output:

     • On each remote machine, set the environment variable MSL_DISTRIBUTED to 1.

     • Alternatively, set this in the component code itself, with the following code: putenv("MSL_DISTRIBUTED=1").

     • The sample components (compose, printer, generator) set it in the code, immediately after the while loop that parses the command line arguments.

   • For the C++ frontend set it according to your choice in samples/components/compose/Compose.cpp for each component.

   • For the the python interface: set the debug level in the python script (run.py)

   ---

5. Running XCAT-C++ with the python front-end.
   Set the path to python and xerces libs by running the following commands:

   • Set XCATCPP_HOME to point to the xcatcpp directory inside xcatcpp-1.0Beta directory

   • setenv PATH ${XCATCPP_HOME}../deps/bin:${PATH}

   • setenv LD_LIBRARY_PATH ${XCATCPP_HOME}../deps/lib
   • Now, run the example

     • cd $(XCATCPP_HOME)/python

     • python run.py -l < PATH_TO_xbmp.so> -d < DEBUG_LEVEL>

     • Example: python run.py -l ../libs/dbg/xct/xbmp.so -d 10

   ---

6. Running XCAT-C++ with the C++ front-end

   • cd $(XCATCPP_HOME)/samples/components/compose

   • ./compose -l < PATH_TO_xbmp.so> -d < DEBUG_LEVEL>

   • Example: ./compose -l ../../../libs/dbg/xct/xbmp.so -d 10

   ---

7. Running XCAT-C++ components from the command-line.

   • If you use SSH to launch components onto remote machines and the application fails, it is hard to detect which component has failed in the component network. XCAT-C++ components can be instantiated from the command line (instead of launching them via SSH). This way the core-dump for each component can be analyzed separately.

   • We explain how to accomplish this using the two sample components that have been provided: printer and generator.

   • Edit printer/makefile to remove PrinterWrapper.cpp and add PrinterWrapperCommandLine.cpp instead for compilation.

   • Edit generator/makefile to remove GeneratorWrapper.cpp and add GeneratorWrapperCommandLine.cpp instead for compilation.

   • Compile the two components (run "make all" inside the component directories).

   • First run the printer: ./printer -l ../../../libs/dbg/xct/xbmp.so -d 100

   • As you are trying to debug, the debug level in the above command has been set to 100, so that the output from all libraries is visble.

   • The printer output will display the reference at which the printer service is available. An example reference is the following: 80e2b650000074ea15a3fa0a4a63003c|xbmp{|80e2b650000074ea0709541f0873eeed:drake:43214|}

   • Now, run the generator component in the following way, using the remote reference of the printer component:

   • ./generator -l ../../../libs/dbg/xct/xbmp.so -d 100 -t "80e2b650000074ea15a3fa0a4a63003c|x bmp{|80e2b650000074ea0709541f0873eeed:drake:43214|}"

   ---

8. Debugging tips for XCAT-C++

   • We recommend that you use the C++ "frontend" for xcat

   • You can load the "compose" component in gdb with this command: gdb --args ./compose -l ../../../libs/dbg/xct/xbmp.so -d 10

   • you'll get a '(gdb)' prompt, where you can type help. The easiest thing to do is just type 'run' and let the program run. If you get a segfault, you should get the '(gdb)' prompt and you can type backtrace.

   • if you want to do more, like code stepping and breakpointing, and become consummate unix hacker, here are a few links to tutorials that looked like they might be helpful.

     • Using gdb to set breakpoints and step through code: http://www.cs.cmu.edu/~gilpin/tutorial/

     • Using Emacs as a 'graphical' frontend to gdb: http://tedlab.mit.edu/~dr/gdbintro.html

     • The full online manual: http://sources.redhat.com/gdb/current/onlinedocs/gdb_toc.html

   ---

9. Writing a new XCAT-Component We explain with reference to two components: "generator" and "printer."

   • The generator component sends different data types (primitives, complex types, and array of primitives) to the "printer" component by invoking methods on its uses ports. It makes blocking calls to the printer component. The printer component receives these calls in its implementation of the provides port.

   • Here is an example of a port definition for the type "AllTypes."

   • XCAT-C++ uses the wsit library for various data types. This helps in using memory management of arrays and complex data types via the smart pointer abstractions provided in wsit. In particular, the AllTypes port uses the following:

     • double[] is represented as wsit::PArrayVobject<double> >

     • double[][] is represented as wsit::MatrixVobject<double> >

   • The printer component has a provides port for AllTypes.

   • The generator component has a uses port for AllTypes.

   • The CCA model has the concept of a go port to bootstrap a network of connected components started with the scientific simulation. The idea is that once a composer (front-end GUI or script), connects all the components together, it can invoke a "go" on one (or many) components to start the simulation. The go port can also be used to send a signal to a component asking it to "kill" itself after a specific number of seconds. Here is the definition of the Go port.

   • In our example, the generator component has a provides port of type "Go."

   • Each component has a wrapper, that contains the main() function. This is useful for remote instantiation of these components. Here is an example of the wrapper for the generator component.

   • The command line arguments are used to successfully instantiate the component:

     • Specifies the path to the communication provider. XCAT-C++ uses a multi-protocol communication substrate and it can dynamically load a library that adheres to the specification for a proteus-provider.

     • It takes the url of a proteus nodepoint for the BuilderService that was used to instantiate this component. Once the component is instantiated by the wrapper, it sends the componentID of the newly instantiated component to the BuilderService.

     • Debug Level: The deseired level of debug statements can be specified as a command line argument for each component.

   • The rest of the responsibilities for the wrapper are the following:

     • Initialize the uses port. This can be done by invoking the init() method in the generated code for each port type. For example see the AllTypeFactory::init() in the GeneratorWrapper code.

     • Create the component and invoke the setServices() method on it. This can be done with the following code:

            // this code is written by the component user
            GeneratorComponent generatorComponent;
            XCATServicesImpl* xcatServicesImpl =
       	 new XCATServicesImpl();
         
            // standard cca call
            generatorComponent.setServices(xcatServicesImpl);
       
       

     • retrieve a componentID of the newly created component.

     • Send the componentID to the BuilderService url that was passed as a command line argument.

   • The component code itself consists of two parts:

     • the setServices() method in which uses and provides ports are added.

     • The core functionality of the component. In the generator component, the core functionality is the executeEcho() method. For the printer component, it just waits for calls on its AllTypes provides port (implemented in ProvidesAllTypesInterfaceImpl.cpp ).

     • Here is the code for the Generator Component and Printer component.

   ---

10. Compiling a new XCAT-C++ component

    • Copy the makefile in component/makefile or printer/makefile and modify it for the new component. In particular, take care of the following:

    • Edit the INCLUDES variable to include the path to any IDL port that is needed by the component.

    • Edit the LIB_PATHS, RUN_PATHS, and IDL_LIBS for the IDL ports used by the component.

    • Edit the list of the source files specified via SRCS.

    ---

11. Compiling a new provider added to xct.

    • . Suppose you copy the directory xct/proteus/providers/xbmp to xct/proteus/providers/xbmp-udt.

    • Make sure that xbmp-udt has a sub-directory named "config" just like xbmp and contains the file "@%.sci.gsfc.nasa.gov"

    • . Edit the file xct/proteus/providers/Target.pb and add xbmp-udt to the list of directories that need to be compiled

    • . Inside the directory xct/proteus, run the following command ../prepbuild/prepbuild create

    • . The above command would have created a directory named "linux-2.4-x86-gcc-3.2.3---32-dyn-dbg" inside xct/proteus

    • In this new directory, run the following command:

    • make clean; make -j4

    ---

12. Setting up SSH access between machines with an empty passphrase. Assume that you need to setup SSH connection between two machines, t2-fe and t3-fe, and want to "ssh" between them without typing a password. The easiest way to accomplish this is to generate your keys on one machine (t3-fe) and copy them to the other machne (t2-fe). Here are the set of commands that you need to run (assuming that the username is john_doe).

    
    [john_doe@t3-fe john_doe]$ ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/john_doe/.ssh/id_rsa):
    /home/john_doe/.ssh/id_rsa already exists.
    Overwrite (y/n)? y
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/john_doe/.ssh/id_rsa.
    Your public key has been saved in /home/john_doe/.ssh/id_rsa.pub.
    The key fingerprint is:
    cc:fa:a7:d3:05:dc:04:61:25:35:06:f5:ef:c8:02:50 john_doe@t3-fe.sci.gsfc.nasa.gov
    [john_doe@t3-fe john_doe]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys
    [john_doe@t3-fe john_doe]$ scp .ssh/id_rsa .ssh/id_rsa.pub t2-fe:.ssh/
    id_rsa                                                                                                                            100%  887     0.9KB/s   00:00
    id_rsa.pub                                                                                                                        100%  238     0.2KB/s   00:00
    [john_doe@t3-fe john_doe]$ ssh t2-fe "cat .ssh/id_rsa.pub >> .ssh/authorized_keys"
    

    [john_doe@t3-fe john_doe]$ chmod 755  .
    
    

    [john_doe@t3-fe john_doe]$ scp -r .ssh/ t2-fe:
    id_rsa                                                                                                                            100%  887     0.9KB/s   00:00
    id_rsa.pub                                                                                                                        100%  238     0.2KB/s   00:00
    authorized_keys                                                                                                                   100% 4067     4.0KB/s   00:00
    known_hosts                                                                                                                       100% 2492     2.4KB/s   00:00
    config                                                                                                                            100%  144     0.1KB/s   00:00
    [john_doe@t3-fe john_doe]$
    
    
    
    

    ---

13. XCAT-C++ Code Structure

    • xcatcpp/bin: has utility script to create a SWIG wrapper for BuilderService.

    • xcatcpp/docs: contains the user guide for xcat-c++

    • xcatcpp/libs: has debug and release versions of libs from proteus that are used by xcat-c++. The libs created by xcat-c++ (for ports and the xcat-c++ library) are also stored in this directory.

    • xcatcpp.python: contains the python script (run.py) that can be used to compose XCAT-C++ applications.

    • xcatcpp/xct_includes: include files from libraries in proteus used by XCAT-C++.

    • xcatcpp/src/xcatcpp: the implementation of the CCA spec. This is the core XCAT-C++ library.

    • xcatcpp/samples/components: example XCAT-C++ components.