{"id":1671,"date":"2023-02-26T18:36:42","date_gmt":"2023-02-26T17:36:42","guid":{"rendered":"https:\/\/iww.inria.fr\/sedgra\/?page_id=1671"},"modified":"2023-03-02T17:42:21","modified_gmt":"2023-03-02T16:42:21","slug":"cluster-de-centre-eng","status":"publish","type":"page","link":"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/","title":{"rendered":"INRIA Grenoble Cluster Access &#8211; Usage"},"content":{"rendered":"<div class=\"fx-toc fx-toc-id-1671\"><h2 class=\"fx-toc-title\">Table of contents<\/h2><ul class='fx-toc-list level-1'>\n\t<li>\n\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#introduction\">Introduction<\/a>\n\t<\/li>\n\t<li>\n\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#getting-started-with-the-inria-grenoble-cluster\">Getting started with the INRIA Grenoble cluster<\/a>\n\t\t<ul class='toc-even level-2'>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#connect-to-the-front-end\">Connect to the front end<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#generate-ssh-keys\">Generate SSH keys<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#ssh-configuration\">SSH configuration<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#data-storage\">Data storage<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#execution-environment\">Execution environment<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#run-your-first-job\">Run your first job<\/a>\n\t\t\t<\/li>\n\t\t<\/ul>\n\t<li>\n\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#for-more-information\">For more information<\/a>\n\t\t<ul class='toc-even level-2'>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#oar\">Oar<\/a>\n\t\t\t<\/li>\n\t\t\t<li>\n\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#singularity-or-apptainer\">Singularity or apptainer<\/a>\n\t\t\t\t<ul class='toc-odd level-3'>\n\t\t\t\t\t<li>\n\t\t\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#docs\">Docs<\/a>\n\t\t\t\t\t<\/li>\n\t\t\t\t\t<li>\n\t\t\t\t\t\t<a href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/#converting-a-docker-image-to-singularity-apptainer\">Converting a Docker image to singularity\/apptainer<\/a>\n\t\t\t\t\t<\/li>\n<\/ul>\n<\/ul>\n<\/ul>\n<\/ul>\n<\/div>\n\n\n<h2><span id=\"introduction\">Introduction<\/span><\/h2>\n<p>The Inria Grenoble centre Cluster can meet limited computing needs. This service is open to Inria users, with priority access to some of the machines for the  research teams that have sponsored their funding.<\/p>\n\n<p>The tools for using the Cluster are:<\/p>\n<ul>\n<li><a href=\"http:\/\/oar.imag.fr\/documentation\/\">OAR<\/a> for computer reservation<\/li>\n<li><a href=\"http:\/\/visu-cp.inrialpes.fr\/monika\">Monika<\/a> for monitoring (see the state of computers)<\/li>\n<li><a href=\"http:\/\/visu-cp.inrialpes.fr\/drawgantt\/\">Drawgantt<\/a> to view the machine usage schedule (be patient) <\/li>\n<li><a href=\"https:\/\/iww.inria.fr\/sedgra\/files\/2022\/09\/HPC.seminaire.singularity.04122018.pdf\">Singularity<\/a> to run an application with your own environment<\/li>\n<li> for simple use-cases, you can use conda in your environment. It is presented in the next section <\/li>\n<\/ul>\n\n<p>The list of machine types is available <a href=\"https:\/\/iww.inria.fr\/sedgra\/files\/2022\/09\/machines_cluster_centre_2022_07-1.pdf\">here<\/a>. The name of the machines are suffixed by the name of the sponsoring team (so priority) otherwise the prefix is <code>cp<\/code><\/p>\n\n<p> The old DSI documentation is <a href=\"https:\/\/doc-si.inria.fr\/display\/SU\/Cluster+de+production+de+Grenoble\">Cluster de Centre<\/a> ( DSI \/ Teams )\n<\/p>\n\n<h2><span id=\"getting-started-with-the-inria-grenoble-cluster\">Getting started with the INRIA Grenoble cluster<\/span><\/h2>\n\n<h3><span id=\"connect-to-the-front-end\">Connect to the front end<\/span><\/h3>\n<p>To be able to connect to the front end you must add an ssh public key to your home directory located at<\/p>\n\n<pre><code>$HOME<\/code><\/pre>\n<p>when you are on bastion or your team infrastructure.<\/p>\n<h3><span id=\"generate-ssh-keys\">Generate SSH keys<\/span><\/h3>\n<p>On your personal machine:<\/p>\n<pre><code>ssh-keygen -t rsa -b 3072 -f ~\/.ssh\/id_rsa<\/code><\/pre>\n<p><strong>Do not leave the passphrase field empty. You need to protect your private key with a password for security reasons.<\/strong><\/p>\n<p>This will generate a private key (<code>~\/.ssh\/id_rsa<\/code>) and a public key (<code>~\/.ssh\/id_rsa.pub<\/code>).<\/p>\n<ul>\n<li>On your workstation at Inria, you need to place the <strong>public<\/strong> key previously generated under <code>\/home\/USERNAME\/.ssh\/id_rsa.pub<\/code>. To do so, you can send your public key to yourself via email or use an USB stick. If you are not physically at Inria, you can contact your <code>system administrators<\/code> to do it for you.<\/li>\n<\/ul>\n<p><strong>Do not send the private key via email nor place it on an USB key ! This is not safe. Only your public key needs to be moved.<\/strong><\/p>\n<pre class=\"wp-block-code\"><code>cat ~\/.ssh\/id_rsa.pub &gt;&gt; ~\/.ssh\/authorized_keys<\/code><\/pre>\n<p>At this point, you should be able to execute the following command on your personal machine to connect to <code>bastion<\/code>.<\/p>\n<pre class=\"wp-block-code\"><code>ssh -i ~\/.ssh\/id_rsa &lt;username&gt;@bastion.inrialpes.fr<\/code><\/pre>\n<h3><span id=\"ssh-configuration\">SSH configuration<\/span><\/h3>\n<p>You can use a ssh configuration file to make it easier for you. To do so, create or update the file <code>$HOME\/.ssh\/config<\/code> with the following lines on you personal machine:<\/p>\n<pre class=\"wp-block-code\"><code>Host bastion \n    HostName bastion.inrialpes.fr\n    User &lt;username&gt;\n    ProxyCommand none\n    IdentityFile ~\/.ssh\/id_rsa \nHost *.inrialpes.fr\n    User &lt;username&gt;\n    ProxyCommand ssh -W %h:22 bastion\n    ForwardX11 yes\n    IdentityFile ~\/.ssh\/id_rsa\nHost access*-cp\n    ProxyCommand ssh &lt;username&gt;@bastion.inrialpes.fr \"\/usr\/bin\/nc %h %p\"\n<\/code><\/pre>\n<p>Replace the occurences of <code>&lt;username&gt;<\/code> with your inria username. You can remove <code>ForwardX11*<\/code> lines if you do not need graphical interface. Now you should be able to connect simply by typing : `<code>ssh bastion<\/code>.<\/p>\n<p>After these steps, you can normally connect to one of the cluster front end. Cluster is accessed from front-ends <tt>access1-cp.inrialpes.fr<\/tt> (Fedora) or <tt>acces2-cp.inrialpes.fr<\/tt>(Ubuntu)<\/p>\n<pre class=\"wp-block-code\"><code>$ ssh access2-cp.inrialpes.fr\n###################################################\n##      BASTION SSH de l'INRIA Rh\u00f4ne-Alpes       ##\n## en cas de probleme merci soumettre un ticket  ##\n##         sur https:\/\/helpdesk.inria.fr         ##\n###################################################\nThe authenticity of host 'access2-cp.inrialpes.fr (&lt;no hostip for proxy command&gt;)' can not be established.\nECDSA key fingerprint is SHA256:hJTGfwFZvC\/8Aw1eQ6KuOjf3IeEbki6jxTsjTLzml48.\nAre you sure you want to continue connecting (yes\/no)? yes\nWarning: Permanently added 'access2-cp.inrialpes.fr' (ECDSA) to the list of known hosts.\nLast login: Tue May 24 14:03:30 2022 from 194.199.18.129\nyour_login@access2-cp:~$ \n<\/code><\/pre>\n\n<h3><span id=\"data-storage\">Data storage<\/span><\/h3>\n<p>There are 2 different storage:<\/p>\n<ul>\n<li>your home directory which has only 10Go of space.<\/li>\n<li>share scratch accessible to all team.<\/li>\n<\/ul>\n<p>You should not store data on your home directory since it has limited space and should be kept for the most important files.<\/p>\n<p>The scratch is available at <strong>\/services\/scratch\/TEAM_NAME\/USERNAME<\/strong><\/p>\n<p>If your team folder does not exist you can create a ticket on helpdesk to create it.<\/p>\n<p>Also if your personal folder does not exist, you should be able to create it, you can also ask to your CMI to create it.<\/p>\n<p>The scratch is the prefered space to store data linked to experiment.<\/p>\n<h3><span id=\"execution-environment\">Execution environment<\/span><\/h3>\n<p>Except if you have specific system dependencies, the recommanded way to run your job on the cluster is to use conda. (Otherwise you should use <a href=\"#singularity\">singularity or apptainer<\/a>. If you need to use it and never used containerization, you should ask for help from someone used to this kind of technology because the learning curve can be really steep.)<\/p>\n<p>To be able to use conda, it is required to install it in <strong>\/services\/scratch\/TEAM_NAME\/USERNAME<\/strong>, it is probably doable to share a conda installation but you will probably need to modify some permissions.<\/p>\n<p>To install conda follow, the following procedure<\/p>\n<ol type=\"1\">\n<li>Go to your home directory or to your scratch directory and download miniconda installer:<\/li>\n<\/ol>\n<pre class=\"wp-block-code\"><code>wget https:\/\/repo.anaconda.com\/miniconda\/Miniconda3-latest-Linux-x86_64.sh<\/code><\/pre>\n<ol type=\"1\" start=\"2\">\n<li>then execute the following command<\/li>\n<\/ol>\n<pre class=\"wp-block-code\"><code>chmod +x Miniconda3-latest-Linux-x86_64.sh &amp;&amp; .\/Miniconda3-latest-Linux-x86_64.sh<\/code><\/pre>\n<ol type=\"1\" start=\"3\">\n<li><p>When asked about the installation location, put it into you scratch space : <strong>\/services\/scratch\/TEAM_NAME\/USERNAME\/miniconda3<\/strong>. Then accept to run conda init.<\/p><\/li>\n<li><p>Source your <strong>~\/.bashrc<\/strong> for the changes to take action : <strong>source ~\/.bashrc<\/strong><\/p><\/li>\n<\/ol>\n<p>Congratulations, now you can create python environments for your projects !<\/p>\n<h3><span id=\"run-your-first-job\">Run your first job<\/span><\/h3>\n<p>The scheduler used to handle the resources is OAR.<\/p>\n<p>You can run a job with the following command:<\/p>\n<pre><code>oarsub -I<\/code><\/pre>\n<p>It will run an interactive job<\/p>\n<p>But there are different parameters which allow to specify the server type<\/p>\n<p>Two kind of parameters exist the ones set with <strong>-l<\/strong> and the ones step with <strong>-p<\/strong><\/p>\n<p>You can specify an host with:<\/p>\n<pre><code>oarsub -I -p \"cluster='thoth' AND host='node3-thoth.inrialpes.fr'\"<\/code><\/pre>\n<p>You have to specify the cluster when targetting a specific node. You can specify multiple node and cluster with a combinaison of <em>OR<\/em> and <em>AND<\/em><\/p>\n<p>You can change the lenght of a job<\/p>\n<pre><code>oarsub -l \"walltime=48:0:0\" \"\/path\/to\/my\/script.sh\" # launch a job for 48 hours<\/code><\/pre>\n<p>Ask for 2 GPU on a single node:<\/p>\n<pre><code>oarsub -l \"\/host=1\/gpudevice=2\"<\/code><\/pre>\n<p>Ask for 2 GPU on 2 nodes:<\/p>\n<pre><code>oarsub -l \"\/host=2\/gpudevice=2\"<\/code><\/pre>\n<p>For example, you can also set, a number of cores, a minimum of RAM \u2026<\/p>\n<p>The name of the parameter correspond to the name of the parameter in monika.<\/p>\n<p>Almost every node have been funded by a specific team, the funding teams are priority on the nodes they have funded and as a consequence can kill every best effort job.<\/p>\n<p>If you are not in a team which has funded nodes, your jobs will be launch by default in the best effort queue and idempotent. Which is the lowest priority queue and idempotent means that your job will be relaunch if it gets killed by a default queue job.<\/p>\n<p>You can connect to your running job with:<\/p>\n<pre><code>oarsub -C **JOB_ID**<\/code><\/pre>\n<p>You can delete your running job with:<\/p>\n<pre><code>oardel **JOB_ID**<\/code><\/pre>\n<p>You can get extra information concerning a job with<\/p>\n<pre><code>oarstat -f -j **JOB_ID**<\/code><\/pre>\n<p>You can show currently running and waiting jobs for a user with<\/p>\n<pre><code>oarstat -u **JOB_ID**<\/code><\/pre>\n<p>Example : Run a script.sh on 4 nodes, using 16 cores on each node, on any cluster, with besteffort, during max 30 minutes:<\/p>\n<pre><code>$ oarsub \u2013l \/nodes=4\/core=16,walltime=00:30:00 -p \"cluster=\u2019SIC\u2019OR cluster=\u2019nanod\u2019 OR cluster=\u2019mistis\u2019\nOR cluster=\u2019kinovis\u2019 OR cluster=\u2019beagle\u2019, OR cluster=\u2019perception\u2019 OR cluster=\u2019thoth\u2019\"\n\/services\/scratch\/morpheo\/kinovis\/script.sh<\/code><\/pre>\n<p>Cluster schedule can be monitored on :<\/p>\n<pre class=\"wp-block-code\"><code>http:\/\/visu-cp.inrialpes.fr\/monika\nhttp:\/\/visu-cp.inrialpes.fr\/drawgantt<\/code><\/pre>\n<p>You need to be connected to the VPN.<\/p>\n\n<h2><span id=\"for-more-information\">For more information<\/span><\/h2>\n\n<p>For a more elaborate use of the cluster, you need to know a bit more about some tools<\/p>\n\n<h3><span id=\"oar\">Oar<\/span><\/h3>\n\n<p>OAR commands:<\/p>\n<table>\n<tbody>\n<tr>\n<td width=\"73\">Cmd<\/td>\n<td width=\"217\">Options<\/td>\n<td width=\"94\">Argument(s)<\/td>\n<td width=\"322\">&nbsp;<\/td>\n<\/tr>\n<tr>\n<td width=\"73\">oarstat<\/td>\n<td width=\"217\">&nbsp;<\/td>\n<td width=\"94\">&nbsp;<\/td>\n<td width=\"322\">shows currently running and waiting jobs, and who they belong to<\/td>\n<\/tr>\n<tr>\n<td rowspan=\"4\" width=\"73\">oarsub<\/td>\n<td width=\"217\">-I<\/td>\n<td rowspan=\"4\" width=\"94\">Script or executable<\/td>\n<td width=\"322\">Interactive mode: connects to a slave node and opens a shell for at most 2 hours<\/td>\n<\/tr>\n<tr>\n<td width=\"217\">-t besteffort<\/td>\n<td width=\"322\">Runs as besteffort, job can be killed by higher privileged job<\/td>\n<\/tr>\n<tr>\n<td width=\"217\">-l \/nodes=x\/cpu=x\/core=x,&nbsp;\n<p>&nbsp;<\/p>\n<p>walltime=xx:xx:xx<\/p>\n<\/td>\n<td width=\"322\">Number of nodes\/cores\/cpus and duration requested for the job<\/td>\n<\/tr>\n<tr>\n<td width=\"217\">-p \u201ccluster=\u2019SIC\u2019[OR cluster=\u2019name\u2019]\u201d<\/td>\n<td width=\"322\">Chose the cluster(s) to run on. Can be SIC, kinovis, nanod, beagle, mistis, perception, thoth. Besteffort on other teams clusters<\/td>\n<\/tr>\n<tr>\n<td width=\"73\">oarsub<\/td>\n<td width=\"217\">-C<\/td>\n<td width=\"94\">JOB_ID<\/td>\n<td width=\"322\">Connects to the job<\/td>\n<\/tr>\n<tr>\n<td width=\"73\">oardel<\/td>\n<td width=\"217\">&nbsp;<\/td>\n<td width=\"94\">JOB_ID<\/td>\n<td width=\"322\">Kills the job<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p>The standard output and error are redirected to OAR.JOB_ID.stdout and OAR.JOB_ID.stderr.<\/p>\n<p>Example : Run a script.sh on 4 nodes, using 16 cores on each node, on any cluster, &nbsp;with besteffort, during max 30 minutes:<\/p>\n\n<pre class=\"wp-block-code\"><code>$ oarsub \u2013l \/nodes=4\/core=16,walltime=00:30:00 -p \"cluster=\u2019SIC\u2019OR cluster=\u2019nanod\u2019 OR cluster=\u2019mistis\u2019 OR cluster=\u2019kinovis\u2019 OR cluster=\u2019beagle\u2019, OR cluster=\u2019perception\u2019 OR cluster=\u2019thoth\u2019\" \/services\/scratch\/morpheo\/kinovis\/script.sh<\/code><\/pre>\n\n<h3><span id=\"singularity-or-apptainer\"><a name=\"singularity\">Singularity or apptainer<\/a><\/span><\/h3>\n\n<h4><span id=\"docs\">Docs<\/span><\/h4>\n<ul>\n<li><a href=\"https:\/\/sed.inrialpes.fr\/docker-tuto\/singularity_20181204.pdf\">Introduction to\ncontainerization with Singularity<\/a> (in English)<\/li>\n<li>A <a href=\"https:\/\/indico.in2p3.fr\/event\/17206\/contributions\/64126\/attachments\/50157\/63972\/Singularity_01.pdf\">tutorial from IN2P3<\/a> to start (in French).<\/li>\n<li><a href=\"http:\/\/apptainer.org\/news\/community-announcement-20211130\">\u00ab\u00a0Singularity has joined the Linux Foundation and is now Apptainer!\u00a0\u00bb<\/a><\/li>\n<\/ul>\n\n<h4><span id=\"converting-a-docker-image-to-singularity-apptainer\">Converting a Docker image to singularity\/apptainer<\/span><\/h4>\n<p>\nTo perform the conversion Singularity 3.0 minimum or apptainer v1 is required.\n\nThere are several ways to convert (run directly a docker image, run directly a docker container but there can be some errors) : the best is building a dedicated singularity image from the docker tar archive.<\/p>\n<p>WARNING: You must be the root user to build from a Singularity recipe file, then singularity must be added to sudoer, with setenv to get the singularity env variables<\/p>\n<pre class=\"wp-block-code\"><code>$ vi \/etc\/sudoers.d\/singularity\nloginx ALL=(root) NOPASSWD:SETENV: \/usr\/local\/bin\/singularity\nloginy ALL=(root) NOPASSWD:SETENV: \/usr\/local\/bin\/singularity\n<\/code><\/pre>\n<pre class=\"wp-block-code\"><code>$ sudo -E singularity build .\/my_image_singularity.img docker-archive:.\/my_image_docker_tar.tar<\/code><\/pre>\n-&gt; \/root is used for cache despite invoking -E with sudo that should tell to use our environment variables !\n<pre class=\"wp-block-code\"><code>$ sudo SINGULARITY_CACHEDIR=\/scratch\/loginx\/singularity\/ SINGULARITY_TMPDIR=\/scratch\/loginx\/singularity\/ singularity build .\/my_image_singularity.img docker-archive:.\/my_image_docker_tar.tar\n<\/code><\/pre>\n<p>FINALLY did the job and created a singularity image from the docker one<\/p>","protected":false},"excerpt":{"rendered":"<p>Introduction The Inria Grenoble centre Cluster can meet limited computing needs. This service is open to Inria users, with priority access to some of the machines for the research teams that have sponsored their funding. The tools for using the Cluster are: OAR for computer reservation Monika for monitoring (see\u2026<\/p>\n<p> <a class=\"continue-reading-link\" href=\"https:\/\/iww.inria.fr\/sedgra\/cluster-de-centre-eng\/\"><span>Continue reading<\/span><i class=\"crycon-right-dir\"><\/i><\/a> <\/p>\n","protected":false},"author":2562,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":"","_members_access_role":[],"_members_access_error":""},"class_list":["post-1671","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/pages\/1671","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/users\/2562"}],"replies":[{"embeddable":true,"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/comments?post=1671"}],"version-history":[{"count":78,"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/pages\/1671\/revisions"}],"predecessor-version":[{"id":1814,"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/pages\/1671\/revisions\/1814"}],"wp:attachment":[{"href":"https:\/\/iww.inria.fr\/sedgra\/wp-json\/wp\/v2\/media?parent=1671"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}