10000 profiling mpi programs is not supported (was: How do use gperftools to profile mpi program?) · Issue #425 · gperftools/gperftools · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

profiling mpi programs is not supported (was: How do use gperftools to profile mpi program?) #425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alk opened this issue Aug 23, 2015 · 5 comments
Open

profiling mpi programs is not supported (was: How do use gperftools to profile mpi program?) #425

alk opened this issue Aug 23, 2015 · 5 comments
Assignees
@alk
Labels
alk-wants-this-soon enhancement Priority-Medium Status-New Type-Defect

Comments

@alk
Copy link
Contributor
alk commented Aug 23, 2015

Originally reported on Google Code with ID 422

What steps will reproduce the problem?
1. mpicc -o t.x t.c -g
2. mpirun_rsh --hostfile=hfile -n 2 LD_PRELOAD=/home/niuq/tools/gperftools-2.0/gcc-build/lib/libtcmalloc.so
HEAPPROFILE=./tprofile ./t.x
3. I can get tprofile.0004.heap  tprofile.0017.heap  tprofile.0030.heap but the thing
is there not each processor seperate heap file. We can not distinguish which processor
heap file comes from?
What is the expected output? What do you see instead?
Expected heap file seperately for each mpi processor. 

What version of the product are you using? On what operating system?
niuq@node020:~/code$ uname -a
Linux node020.cluster 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011
x86_64 x86_64 x86_64 GNU/Linux
gperftools-2.0


Please provide any additional information below.

niuq@node020:~/code$ cat t.c
#include<stdio.h>
#include<stdlib.h>
#include<time.h>
#include <string.h> 
#include <mpi.h> 
int main(int argc, char *argv[]) { 
    MPI_Init(&argc,&argv);
    int myrank;
    MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
    if(myrank==1)
    {
        size_t x=sizeof(int);
        printf( "size = %lu\n", x ); 
        int i;
        for(i=0;i<10000000;i++)
        {
            unsigned long s=100;
            int* sum=(int*)malloc(s*sizeof(int));
            int k;
            for(k=1;k<s;k++) sum[k]=sum[k-1]+5;
        }
    }
    MPI_Finalize();
    return 0;
} 

mpicc -o t.x t.c -g

Reported by niuqingpeng on 2012-04-02 18:35:48
@alk alk self-assigned this Aug 23, 2015
@alk
Copy link
Contributor Author
alk commented Aug 23, 2015 
Waiting on further comment. We had a possible solution over an email chat but it turns
out that it doesn't cover all MPI implementations.

Reported by chappedm on 2012-05-04 12:51:28

@alk
Copy link
Contributor Author
alk commented Aug 23, 2015 
The attached patch includes the "MPI rank" in the output filename, at least for
the openmpi MPI implementation.

Background: MPI provides methods for parallel computing beneath
multithreading, i.e. parallel processing without shared memory. A particular
job may run on many processors distributed among many nodes simultanously,
therefore getpid() does not provide a unique id.
In the attached patch, I include the MPI rank (in the openmpi MPI
implementation) explicitely when generating file names for the profiler.

A method independent of the specific MPI implementation would require explicit calls
to the MPI library in order to obtain the process rank, which in turn requires initializing
MPI. This has two disadvantages: for one, libtcmalloc would then depend on MPI (which
is not that common outside the high-performance computing (HPC) context); for another,
interacting with the MPI implementation from within libtcmalloc will violate assumptions
made by the program being debugged.

My patch evaluates an environment variable defined by the openmpi implementation, which
according to their FAQ [1] is guaranteed to be stable in future releases.
The patch is written in such a way that it is easy to add environment variables related
to other MPI implementations; intel mpi provides $PMI_RANK [2].
I implemented only for openmpi, as thats the implementation I can test on our HPC cluster.

[1] http://www.open-mpi.org/faq/?category=running#mpi-environmental-variables
[2] https://software.intel.com/de-de/forums/topic/284007

--
c u
henning

Reported by henning.glawe on 2015-08-04 06:35:41

- _Attachment: [include_MPI_rank_in_filename__if_applicable.patch](https://storage.googleapis.com/google-code-attachments/gperftools/issue-422/comment-2/include_MPI_rank_in_filename__if_applicable.patch)_

@alk alk changed the title How do use gperftools to profile mpi program? profiling mpi programs is not supported (was: How do use gperftools to profile mpi program?) Feb 22, 2016
@alk
Copy link
Contributor Author
alk commented Jun 26, 2016

This seems to be same problem as issue #601. I.e. if program forks but doesn't exec (as mpi implementations might do) all instances will use same file.

@satishskamath
Copy link

@alk I cannot access this patch or is it already a part of gperftools. Also does this work for CPUPROFILE?

@alk
Copy link
Contributor Author
alk commented Jul 23, 2023

#1263 is in flight right now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alk alk

Labels
alk-wants-this-soon enhancement Priority-Medium Status-New Type-Defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alk @satishskamath
0