In this blog post, I will show you an example of the creation of a dtk base application : dtk-traces .

In case you don’t know it already, dtk is a meta-platform to develop scientific applications. A thoughtful presentation can be found here: dtk-introduction

Traces

First of all, let’s talk a bit about TRACES. It is a program made by ANDRA to perform numerical simulations of radioactive waste storage in deep geological layers.

It is used by ANDRA to study the transportation of radionucleides as it’s able to handle characteristic phenomenons of this model such as:

• convection (movement carried by water flow)
• cinematic dispersion and molecular dispersion (to homogenise concentration)
• adsorption
• precipitation/dissolution
• degradation

It’s coded in Fortran, parallelised using MPI, can be compiled either in a binary or in a dynamic library, and follows a simple 4-steps workflow:

 

dtkDistributedApplication

The first step to do for using our Fortran library with a dtk-based application is to use dtkDistributedApplication class. This class is a specialization of QApplication providing standard arguments and uniformizing applications based on our platform. It simplifies arguments handling, MPI initialisation and calls, the creation of a main, and the execution of a program in parallel. You can get a full example of a dtk application here.
Below is a basic example of an application based on this class:

int main(int argc, char **argv)
{
  dtkDistributedApplication *app = dtkDistributed::create(argc, argv);
  app->setOrganizationName("inria");
  app->setOrganizationDomain("fr");
  app->setApplicationName("dtkTraces");
  app->setApplicationVersion("0.1.0");

  QCommandLineParser *parser = app->parser();
  parser->setApplicationDescription("dtk wrapper of Traces.");

  QCommandLineOption meshFileOption("mesh_file", "mesh_file (full path to the file)", "mesh_file");
  parser->addOption(meshFileOption);

  app->initialize();

  // ------------ check parameters
  if (!parser->isSet(meshFileOption))  {
    qFatal("Error: input are not correctly set! you have to set : --mesh_file <filename>") ;
    return 1;
  }

  // /////////////////////////////////////////////////////////////////
  // Launch the execution
  // ////////////////////////////////////////////////////////////////
  dtkTraces traces = dtkTraces();

  traces.mesh_file = parser->value(meshFileOption);
 
  app->spawn();
  app->exec(&traces);
  app->unspawn(); // unspawn will quit the application
  delete app;
  return 0;
}

dtkTraces

The second step we have to do is to create the dtkTraces class that will be executed! Indeed, in the main example that I just showed, the following 3 lines may have catch your eyes :

  app->spawn();
  app->exec(&traces);
  app->unspawn(); 

dtkDistributedApplication can simplify the execution of a program in parallel for you by spawning itself (for a mpi implementation, it relies on MPI_Comm_spawn ).
In order to do so, the program to be executed in parallel need to be in a class derived from QRunnable. For us, it is dtkTraces.

The only constraint coming with it is giving an implementation of the pure virtual function void QRunnable::run() in your subclass as this:

void dtkTraces::run(void)
{
    QTime timer;

    dtkDistributedCommunicator *comm = dtkDistributed::communicator::instance();
    int mpi_rank = comm->rank();
    int mpi_size = comm->size();

    // I call my fortran subroutine here

 

 
Note That current supported schedulers for spawn are TORQUE and OAR. If you are executing your program on a cluster with another scheduler (SLURM for example), run your application using mpirun or srun and use the argument –no-spawn at runtime.

Another striking element, may be:

  dtkDistributedCommunicator *comm = dtkDistributed::communicator::instance();

When I said that dtkDistributedApplication simplify MPI calls, I meant that app->initialize() will create and initialise a dtkDistributedCommunicator for us based on the policy argument given at runtime. For example, if you specify –policy mpi3 it will create a MPI_Comm communicator as well as shared communicator for intra-node messages (based on MPI_comm_split). You can then use optimised functions such as send, receive, reduce, addAssign, compareAndSwap, .. .

Of course, if you want to get the underlying communicator from dtkDistributedCommunicator, you easily can. If you know it’s a mpi communicator, use this code:

MPI_Comm *mpi_comm = static_cast<MPI_Comm *>(comm->data());

command line

As I said, dtkDistributedApplication come with defaults arguments. Here they are:

• –help display the default arguments
• –policy <qthread|mpi|mpi3> dtkDistributedPolicy (default is qthread)
• –nw, –no-window non GUI applications
• –loglevel <trace|debug|info|warn|error|fatal> loglevel to write
• –logfile <file_name|console> file_name to write logs int
• –no-spawn to use if you are directly launching your application with mpirun. It is the preferred method for clusters
• –np X to spawn X processes from the master thread. Unused when –no-spawn is present

Lastly, here is the final command line corresponding to the main presented above:

• spawn ./dtkTracesApp --np 2 --policy mpi3 --mesh_file /path/to/my/mesh_file --logfile console --loglevel debug --nw
• no spawn mpirun -np 2 ./dtkTracesApp --policy mpi3 --mesh_file /path/to/my/mesh_file --logfile console --loglevel debug --no-spawn --nw