Python Numpy Memory Leaks Due to PYTHONMALLOC (C Python and Address Sanitizer)

A simple C/C++ calling Python script to import numpy package will simply be reported Leaking Memory:

1
2
3
4
5
6
7
8

include <Python.h>
 
int main() {
    Py_Initialize();
    PyRun_SimpleString("import numpy");
    Py_Finalize();
    return 0;
}

include <Python.h>

int main() {
    Py_Initialize();
    PyRun_SimpleString("import numpy");
    Py_Finalize();
    return 0;
}

This will be caught leaking memory by Address Sanitizer which is a tool to inspect the potential memory leaks and other memory issues such as Heap Overflow, Double Free, Heap Use after Free in the code.

==7==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 412 byte(s) in 38 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2d19 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cd19)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3218 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d218)

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3200 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d200)

Direct leak of 23 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2d58 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cd58)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131c3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7fa32a6d3233 in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39d233)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2dbe in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cdbe)

Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x7fa330131808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7fa32a6d2dad in PyUFunc_FromFuncAndDataAndSignatureAndIdentity (/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so+0x39cdad)

SUMMARY: AddressSanitizer: 519 byte(s) leaked in 44 allocation(s).

Python uses the default memory allocator, which seems to be causing problems issue. Fortunately, we can replace it by setting Environmental Variable PYTHONMALLOC to malloc – the C library version.

1

export PYTHONMALLOC=malloc

export PYTHONMALLOC=malloc

“import numpy” is reported as memory leaks by Address Sanitizer. How can I ignore this?

Address Sanitizer (ASan) is a tool used to detect memory errors such as buffer overflow, use-after-free, and memory leaks. Memory leaks occur when memory is allocated but not freed, leading to a gradual accumulation of memory usage over time.

If you have determined that the memory leak is not a critical issue and you want to ignore it, you can use ASan’s suppression mechanism to ignore the leak. To do this, you need to create a suppression file that specifies the functions or modules to ignore.

Here’s an example suppression file that ignores memory leaks in the libpython (which is Python module for C/C++):

1
2

# Ignore memory leaks in the Python for C Module
leak:libpython

# Ignore memory leaks in the Python for C Module
leak:libpython

To use this suppression file with ASan, you can pass the path to the file as a command-line argument to your program, like this:

1
2

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so ASAN_OPTIONS=detect_leaks=1 \
   ASAN_OPTIONS=suppressions=/path/to/suppression/file ./your_program

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so ASAN_OPTIONS=detect_leaks=1 \
   ASAN_OPTIONS=suppressions=/path/to/suppression/file ./your_program

This command sets the ASAN_OPTIONS environment variable to enable leak detection and load the ASan library, and then specifies the suppression file to use with the suppressions option.

Note that ignoring memory leaks is generally not recommended as it can lead to a gradual buildup of memory usage, which can eventually cause performance issues or even crashes. It’s usually better to identify and fix the root cause of the leak if possible.

–EOF (The Ultimate Computing & Technology Blog) —

GD Star Rating
loading…

875 words
Last Post: How ChatGPT Solves a Math Puzzle?