Ошибка out of memory matlab

When I run a sample script in MATLAB, it says:

Out of memory. Type HELP MEMORY for your options.

When I type «memory», it reports:

Maximum possible array:             156 MB (1.638e+008 bytes) *
Memory available for all arrays:    740 MB (7.756e+008 bytes) **
Memory used by MATLAB:             1054 MB (1.105e+009 bytes)
Physical Memory (RAM):             3070 MB (3.219e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

Is there any way to get around this error? I’m using Windows XP x32 with MATLAB 2009a.

Peter Mortensen's user avatar

asked Aug 14, 2009 at 11:08

Contango's user avatar

2

pack does a memory defragmentation. It might help you a bit as far as the contiguous memory available.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 11:39

Marcin's user avatar

MarcinMarcin

3,4381 gold badge22 silver badges15 bronze badges

0

Remember, when MATLAB says it’s out of memory, it means it’s out of contiguous memory, so rebooting or restarting MATLAB may work.

But, I’d recommend optimizing your code and identifying how you’re eating up so much memory. It could be an ill-designed recursive loop, or a bad indexing function (using doubles instead of logicals to index a huge matrix).

I practically lived with memory errors for a while since I was dealing with huge datasets, but there’s always a workaround, ask specific questions and you’ll be surprised.

Community's user avatar

answered Aug 14, 2009 at 12:19

Jacob's user avatar

JacobJacob

34.1k14 gold badges110 silver badges165 bronze badges

Problem fixed.

Under Windows XP x32, I managed to almost double the amount of memory available to MATLAB by editing boot.ini to add the switch /3GB /USERVA=3030

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB /USERVA=3030

Together with reducing our array sizes, this completely fixed the problem :)

I could have also fixed the problem by upgrading to Windows x64 or Windows 7 x64. This act also doubles the amount of memory available to MATLAB, even if you stick with MATLAB x32 and don’t upgrade to MATLAB x64. Windows x64 is just far more memory efficient, even with systems that only have 4 GB of physical RAM installed.

Peter Mortensen's user avatar

answered Aug 17, 2009 at 11:08

Contango's user avatar

ContangoContango

75.9k57 gold badges259 silver badges302 bronze badges

5

Try this, it works well for me.

  • Go to Home -> Preference icon -> General -> Java Heap Memory -> Allocate what size of memory you want
  • In Preference window, go to «Workspace» (out of Java heap memory level) -> See «Matlab Array size limit»
    Make sure uncheck the ‘Limit the maximum array size to a percentage of RAM’. Because you want to extend memory
    so we don’t need this feature.
  • Done.

answered Mar 22, 2017 at 16:04

Vinh Trieu's user avatar

Vinh TrieuVinh Trieu

9938 silver badges12 bronze badges

What are you attempting to allocate when it runs out of memory (OOM)? Do you have code to reproduce? A wide range of problems can cause out of memory errors.

To diagnose, use «dbstop if all error» to set a breakpoint on errors. The out of memory will trigger this, and you can use dbup, dbdown, and whos() to see what’s consuming memory. Often an OOM is caused by a bad array size or index calculation, not just by big data structures. E.g. this will trigger an OOM in pretty much any 32-bit MATLAB.

>> x = 1;
>> x(2^30) = 2
??? Out of memory. Type HELP MEMORY for your options.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 16:15

Andrew Janke's user avatar

Andrew JankeAndrew Janke

23.5k5 gold badges56 silver badges85 bronze badges

0

I faced a similar error while running an (old) C file in MATLAB using mex.

I found my solution at this issue on GitLab.

First, uncheck the option «Limit the maximum Array Size to a % of RAM» located under Preferences -> Workspace, as also indicated in this earlier answer.

Once applied, run your C file in the command window using

mex filename.c -compatibleArrayDims

Cris Luengo's user avatar

Cris Luengo

55k10 gold badges61 silver badges119 bronze badges

answered Sep 14, 2020 at 10:40

VIREN RAMCHANDANI's user avatar

4

Issue

When your code operates on large amounts of data or does not use memory
efficiently, MATLAB® might produce an error in response to an unreasonable array size, or
it might run out of memory. MATLAB has built-in protection against creating arrays that are too large.
For example, this code results in an error, because MATLAB cannot create an array with the requested number of elements.

Requested array exceeds the maximum possible variable size.

By default, MATLAB can use up to 100% of the RAM (not including virtual memory) of your
computer to allocate memory for arrays, and if an array size would exceed that
threshold, then MATLAB produces an error. For example, this code attempts to create an array
whose size exceeds the maximum array size limit.

Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size preference (63.7GB). This might cause MATLAB to become
unresponsive.

If you turn off the array size limit in MATLAB Workspace
Preferences
, attempting to create an unreasonably large array might
cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory paging
(that is, moving memory pages between RAM and disk).

Possible Solutions

No matter how you run into memory limits, MATLAB provides several solutions depending on your situation and goals. For
example, you can improve the way your code uses memory, leverage specialized data
structures such as datastores and tall arrays, take advantage of pooled resources
within a computing cluster, or make adjustments to your settings and
preferences.

Note

The solutions presented here are specific to MATLAB. To optimize system-wide memory performance, consider adding more
physical memory (RAM) to your computer or making adjustments at the operating
system level.

Clear Variables When No Longer Needed

Make it a practice to clear variables when they are no longer needed. To clear
items from memory, use the clear function.

Before After
A = rand(1e4);
disp(max(A,[],"all"))
B = rand(1e4);
A = rand(1e4);
disp(max(A,[],"all"))
clear A
B = rand(1e4);

For more information, see Strategies for Efficient Use of Memory.

Use Appropriate Data Storage

Memory requirements differ for MATLAB data types. You might be able to reduce the amount of memory used
by your code by using the appropriate data type and storage. For more
information about the solutions in this section, see Strategies for Efficient Use of Memory.

Use the Appropriate Numeric Class.  The numeric class you should use depends on your intended actions. In
MATLAB, double is the default numeric data type
and provides sufficient precision for most computational tasks:

  • If you want to perform complicated math such as linear algebra,
    use floating-point numbers in either double-precision
    (double) or single-precision
    (single) format. Numbers of type
    single require less memory than numbers of
    type double, but are also represented to less
    precision.

  • If you just need to carry out simple arithmetic and you represent
    the original data as integers, use the integer classes in
    MATLAB.

Class (Data Type) Bytes Supported Operations
single 4 Most math
double 8 All math
logical 1 Logical/conditional operations
int8, uint8 1 Arithmetic and some simple functions
int16, uint16 2 Arithmetic and some simple functions
int32, uint32 4 Arithmetic and some simple functions
int64, uint64 8 Arithmetic and some simple functions

Reduce the Amount of Overhead When Storing Data.  When you create a numeric or character array, MATLAB allocates a block of memory to store the array data.
MATLAB also stores information about the array data, such as its
class and dimensions, in a small, separate block of memory called a
header. Because simple numeric and character arrays
have the least overhead, use them whenever possible. Use other data
structures only for data that is too complex to store in a simple
array.

For structures and cell arrays, MATLAB creates a header not only for the array, but also for each
field of the structure or each cell of the cell array. Therefore, the amount
of memory required to store a structure or cell array depends not only on
how much data it holds, but also on how it is constructed. As a result, cell
arrays with many small elements or structures with many fields containing
little content have large overhead and should be avoided.

Before After
% S has 15,000 fields (3 fields per array element)
for i = 1:100
    for j = 1:50
        S(i,j).R = 0;  % Each field contains a numeric scalar
        S(i,j).G = 0;
        S(i,j).B = 0;
    end
end
% S has 3 fields 
S.R = zeros(100,50);  % Each field contains a numeric array
S.G = zeros(100,50);
S.B = zeros(100,50);

Make Arrays Sparse When Possible.  A good practice is to store matrices with few nonzero elements using
sparse storage. When a full matrix has a small number of nonzero elements,
converting the matrix to sparse storage typically improves memory usage and
code execution time. MATLAB has several functions that support sparse storage. For
example, you can use the speye function to create a
sparse identity matrix.

Before After
I = eye(1000);
I = speye(1000);

Import Data Using the Appropriate MATLAB Class.  When reading data from a binary file with fread, a common mistake is
to specify only the class of the data in the file and not the class of the
data MATLAB uses once it is in the workspace. If you do not specify the
class of the data in memory, MATLAB uses double even if you read 8-bit
values.

Before After
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8"); 
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8=>uint8"); 

Avoid Unnecessary Copies of Data

To improve memory usage and execution speed, make sure your code does not
result in unnecessary copies of data. For more information about the solutions
in this section, see Avoid Unnecessary Copies of Data and Strategies for Efficient Use of Memory.

Avoid Creating Temporary Arrays.  Avoid creating temporary arrays when they are not necessary. For example,
instead of creating an array of zeros as a temporary variable and then
passing that variable to a function, use one command to do both
operations.

Before After
A = zeros(1e6,1);
As = single(A);
As = zeros(1e6,1,"single");

Preallocate Memory.  When you work with large data sets, repeatedly resizing arrays might cause
your program to run out of memory. If you expand an array beyond the
available contiguous memory of its original location, MATLAB must make a copy of the array and move the copy into a memory
block with sufficient space. During this process, two copies of the original
array exist in memory. You can improve the memory usage and code execution
time by preallocating the maximum amount of space required for the array.
For more information, see Preallocation.

Before After
x = 0;
for k = 2:1000000
   x(k) = x(k-1) + 5;
end
x = zeros(1,1000000);
for k = 2:1000000
   x(k) = x(k-1) + 5;
end

Use Nested Functions to Pass Fewer Arguments.  When calling a function, MATLAB typically makes a temporary copy of the variable in the
caller’s workspace if the function modifies its value. MATLAB applies various techniques to avoid making unnecessary copies,
but avoiding a temporary copy of an input variable is not always
possible.

One way to avoid temporary copies in function calls is to use nested
functions. A nested function shares the workspace of all outer functions, so
you do not need to pass a copy of variables in the function call. For more
information, see Nested Functions.

Load Only as Much Data as You Need

One way to fix memory issues is to import into MATLAB only as much of a large data set as you need for the problem you
are trying to solve. Data set size is not usually a problem when importing from
sources such as a database, where you can explicitly search for elements
matching a query. But it is a common problem with loading large flat text or
binary files.

The datastore function lets you
work with large data sets incrementally. Datastores are useful any time you want
to load only some portions of a data set into memory at a time.

To create a datastore, provide the name of a file or a directory containing a
collection of files with similar formatting. For example, with a single file,
use the
following.

ds = datastore("path/to/file.csv");

Or
with a collection of files in a folder, use the
following.

ds = datastore("path/to/folder/");

You
also can use the wildcard character * to select all files of
a specific
type.

ds = datastore("path/to/*.csv");

Datastores
support a wide variety of file formats (tabular data, images, spreadsheets, and
so on). For more information, see Select Datastore for File Format or Application.

Aside from datastores, MATLAB also has several other functions to load parts of files. This
table summarizes the functions by the type of files they operate on.

File Type Partial Loading
MAT-file

Load part of a variable by indexing into an object that
you create with the matfile
function. For more information, see Save and Load Parts of Variables in MAT-Files.

Text

Use the textscan
function to access parts of a large text file by reading
only the selected columns and rows. If you specify the
number of rows or a repeat format number with
textscan, MATLAB calculates the exact amount of memory required
beforehand.

Binary

You can use low-level binary file I/O functions, such
as fread, to
access parts of any file that has a known format. For binary
files of an unknown format, try using memory mapping with
the memmapfile
function.

Image, Audio, Video, and HDF

Many of the MATLAB functions that support loading from these
types of files allow you to select portions of the data to
read. For details, see the function reference pages listed
in Supported File Formats for Import and Export.

Use Tall Arrays

Tall arrays help you work with data sets that are too large to fit in memory.
MATLAB works with small blocks of the data at a time, automatically
handling all of the data chunking and processing in the background. You can use
tall arrays in two primary ways:

  • If you have a large array that fits in memory, but you run out of
    memory when you try to perform calculations, you can cast the array
    to a tall
    array.

    This
    approach lets you work with large arrays that can fit in memory, but
    that consume too much memory to allow for copies of the data during
    calculations. For example, if you have 8 GB of RAM and a 5 GB
    matrix, casting the matrix to a tall array enables you to perform
    calculations on the matrix without running out of memory. For an
    example of this usage, see tall.

  • If you have file- or folder-based data, you can create a datastore
    and then create a tall array on top of the
    datastore.

    ds = datastore("path/to/file.csv");
    t = tall(ds);

    This
    approach gives you the full power of tall arrays in MATLAB. The data can have any number of rows and MATLAB does not run out of memory. And because
    datastore works with both local and remote
    data locations, the data you work with does not need to be on the
    computer you use to analyze it. See Work with Remote Data for more
    information.

To learn more about tall arrays, see Tall Arrays for Out-of-Memory Data.

Use Memory of Multiple Machines

If you have a cluster of computers, you can use distributed arrays (requires
Parallel Computing Toolbox™) to perform calculations using the combined memory of all the
machines in the cluster. Depending on how your data fits in memory, different
ways to partition data among workers of a parallel pool exist. For more
information, see Distributing Arrays to Parallel Workers (Parallel Computing Toolbox).

Adjust Settings and Preferences

Generally, rewriting code is the most effective way to improve memory
performance. However, if you cannot make changes to your code, these solutions
might provide it with the required amount of memory.

Start MATLAB Without Java Virtual Machine or Decrease the Java Heap Size.  If you either start MATLAB without the Java® Virtual Machine (JVM™) software or decrease the Java heap size, you can increase the available workspace memory. To
start MATLAB without JVM, use the command-line option -nojvm. For
information on how to decrease the Java heap size, see Java Heap Memory Preferences.

Using -nojvm comes with a penalty in that you lose
several features that rely on JVM, such as the desktop tools and graphics. Starting MATLAB with the -nodesktop option does not save
any substantial amount of memory.

Adjust the Array Size Limit.  If you face an error stating that the array size exceeds the maximum array
size preference, you can adjust this array size limit in MATLAB. For information on adjusting the array size limit, see Workspace and Variable Preferences. This solution is helpful only if the
array you want to create exceeds the current maximum array size limit but is
not too large to fit in memory. Even if you turn off the array size limit,
attempting to create an unreasonably large array might cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory
paging.

See Also

memory | whos | datastore | tall

Related Topics

  • How MATLAB Allocates Memory
  • Strategies for Efficient Use of Memory
  • Avoid Unnecessary Copies of Data
  • Large Files and Big Data

When I run a sample script in MATLAB, it says:

Out of memory. Type HELP MEMORY for your options.

When I type «memory», it reports:

Maximum possible array:             156 MB (1.638e+008 bytes) *
Memory available for all arrays:    740 MB (7.756e+008 bytes) **
Memory used by MATLAB:             1054 MB (1.105e+009 bytes)
Physical Memory (RAM):             3070 MB (3.219e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

Is there any way to get around this error? I’m using Windows XP x32 with MATLAB 2009a.

Peter Mortensen's user avatar

asked Aug 14, 2009 at 11:08

Contango's user avatar

2

pack does a memory defragmentation. It might help you a bit as far as the contiguous memory available.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 11:39

Marcin's user avatar

MarcinMarcin

3,4381 gold badge22 silver badges15 bronze badges

0

Remember, when MATLAB says it’s out of memory, it means it’s out of contiguous memory, so rebooting or restarting MATLAB may work.

But, I’d recommend optimizing your code and identifying how you’re eating up so much memory. It could be an ill-designed recursive loop, or a bad indexing function (using doubles instead of logicals to index a huge matrix).

I practically lived with memory errors for a while since I was dealing with huge datasets, but there’s always a workaround, ask specific questions and you’ll be surprised.

Community's user avatar

answered Aug 14, 2009 at 12:19

Jacob's user avatar

JacobJacob

34.1k14 gold badges110 silver badges165 bronze badges

Problem fixed.

Under Windows XP x32, I managed to almost double the amount of memory available to MATLAB by editing boot.ini to add the switch /3GB /USERVA=3030

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB /USERVA=3030

Together with reducing our array sizes, this completely fixed the problem :)

I could have also fixed the problem by upgrading to Windows x64 or Windows 7 x64. This act also doubles the amount of memory available to MATLAB, even if you stick with MATLAB x32 and don’t upgrade to MATLAB x64. Windows x64 is just far more memory efficient, even with systems that only have 4 GB of physical RAM installed.

Peter Mortensen's user avatar

answered Aug 17, 2009 at 11:08

Contango's user avatar

ContangoContango

75.9k57 gold badges259 silver badges302 bronze badges

5

Try this, it works well for me.

  • Go to Home -> Preference icon -> General -> Java Heap Memory -> Allocate what size of memory you want
  • In Preference window, go to «Workspace» (out of Java heap memory level) -> See «Matlab Array size limit»
    Make sure uncheck the ‘Limit the maximum array size to a percentage of RAM’. Because you want to extend memory
    so we don’t need this feature.
  • Done.

answered Mar 22, 2017 at 16:04

Vinh Trieu's user avatar

Vinh TrieuVinh Trieu

9938 silver badges12 bronze badges

What are you attempting to allocate when it runs out of memory (OOM)? Do you have code to reproduce? A wide range of problems can cause out of memory errors.

To diagnose, use «dbstop if all error» to set a breakpoint on errors. The out of memory will trigger this, and you can use dbup, dbdown, and whos() to see what’s consuming memory. Often an OOM is caused by a bad array size or index calculation, not just by big data structures. E.g. this will trigger an OOM in pretty much any 32-bit MATLAB.

>> x = 1;
>> x(2^30) = 2
??? Out of memory. Type HELP MEMORY for your options.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 16:15

Andrew Janke's user avatar

Andrew JankeAndrew Janke

23.5k5 gold badges56 silver badges85 bronze badges

0

I faced a similar error while running an (old) C file in MATLAB using mex.

I found my solution at this issue on GitLab.

First, uncheck the option «Limit the maximum Array Size to a % of RAM» located under Preferences -> Workspace, as also indicated in this earlier answer.

Once applied, run your C file in the command window using

mex filename.c -compatibleArrayDims

Cris Luengo's user avatar

Cris Luengo

55k10 gold badges61 silver badges119 bronze badges

answered Sep 14, 2020 at 10:40

VIREN RAMCHANDANI's user avatar

4

Решение “Из памяти” ошибок

Эта тема объясняет несколько стратегий, которые можно использовать в ситуациях, где MATLAB® исчерпывает память. MATLAB является 64-битным приложением, которое работает на 64-битных операционных системах. Это возвращает сообщение об ошибке каждый раз, когда это запрашивает сегмент памяти от операционной системы, которая больше, чем, что доступно.

MATLAB имеет встроенную защиту от создания массивов, которые являются слишком большими. По умолчанию MATLAB может использовать до 100% RAM (не включая виртуальную память) вашего компьютера, чтобы выделить память для массивов, и если массив превысил бы тот порог, то MATLAB возвращает ошибку. Например, этот оператор пытается создать массив с неблагоразумным размером:

Error using rand
Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size
preference. Creation of arrays greater than this limit may take a long
time and cause MATLAB to become unresponsive. See array size limit or
preference panel for more information.

Смотрите Рабочую область и Настройки переменной для получения информации о корректировке этого предела размера массивов. Если вы выключаете предел размера массивов, то MATLAB возвращает различную ошибку:

Out of memory. Type "help memory" for your options.

Неважно, как вы сталкиваетесь с пределами памяти, существует несколько разрешений, доступных в зависимости от ваших целей. Методы, обсужденные в Стратегиях Эффективного использования Памяти, могут помочь вам оптимизировать доступную память, включая которую вы имеете:

  • Используйте соответствующее хранение данных

  • Предотвращение создавать Временные массивы

  • Исправьте используемую память

Если вы уже используете память эффективно, и проблема сохраняется, то остающиеся разделы этой страницы содержат возможные решения.

Усильте tall Массивы

Длинные массивы для Данных, которые не помещаются в память, спроектированы, чтобы помочь вам работать с наборами данных, которые являются слишком большими, чтобы поместиться в память. MATLAB работает с маленькими блоками данных за один раз, автоматически обрабатывая все разделение на блоки данных и обработку в фоновом режиме. Существует два первичных способа, которыми можно усилить длинные массивы:

  1. Если у вас есть большой массив, который умещается в памяти, но у вас заканчивается память, когда вы пытаетесь выполнить вычисления, можно бросить массив к длинному массиву:

    Этот метод включает вам, работают с большими массивами, которые могут уместиться в памяти, но которые используют слишком много памяти, чтобы допускать копии данных во время вычислений. Например, если у вас есть 8 ГБ RAM, и матрица на 5 ГБ, бросая матрицу к длинному массиву позволяет вам выполнить вычисления на матрице, не исчерпывая память. Смотрите Преобразуют Массивы В оперативной памяти в Длинные массивы для примера этого использования.

  2. Если у вас есть файл или основанные на папке данные, можно создать datastore и затем создайте высокий массив поверх datastore:

    ds = datastore('path/to/data.csv');
    tt = tall(ds);

    Этот метод дает вам полную мощность длинных массивов в MATLAB: данные могут иметь любое количество строк, и в MATLAB не заканчивается память. И потому что datastore работает и с районами локальных и с удаленных данных, данные, с которыми вы работаете, не должны быть на компьютере, который вы используете, чтобы анализировать их. Смотрите работу с Удаленными данными для получения дополнительной информации.

Усильте память о нескольких машинах

Если у вас есть кластер компьютеров, можно использовать Parallel Computing Toolbox™ и Распределенные Массивы (Parallel Computing Toolbox), чтобы выполнить вычисления с помощью объединенной памяти обо всех машинах в кластере. Это позволяет вам работать с целым распределенным массивом как одна сущность. Однако рабочие действуют только с их стороны массива, и автоматически передают данные между собой при необходимости.

Создание распределенного массива очень похоже на создание длинного массива:

ds = datastore('path/to/data.csv');
dt = distributed(ds);

Загрузите только столько данные, сколько вам нужно

Другой возможный способ зафиксировать проблемы памяти состоит в том, чтобы только импортировать в MATLAB такое количество большого набора данных, как вам нужно для задачи, которую вы пытаетесь решить. Это обычно не проблема при импорте из источников, таких как база данных, где можно явным образом искать элементы, совпадающие с запросом. Но это — типичная проблема с загрузкой большого плоского текста или двоичных файлов.

datastore функция позволяет вам работать с большими наборами данных инкрементно. Эта функция подкрепляет Длинные массивы для Данных, которые не помещаются в память, и Распределенных Массивов (Parallel Computing Toolbox), но можно использовать его для других целей также. Хранилища данных полезны любое время, вы хотите загрузить небольшие части набора данных в память за один раз.

Чтобы создать datastore, необходимо обеспечить имя файла или директории, содержащей набор файлов с подобным форматированием. Например, с одним файлом:

ds = datastore('path/to/file.csv')

Или, с набором файлов в папке:

ds = datastore('path/to/folder/')

Можно также использовать подстановочный символ * выбрать все файлы определенного типа, как в:

ds = datastore('data/*.csv')

Хранилища данных поддерживают большое разнообразие форматов общего файла (табличные данные, изображения, электронные таблицы, и так далее). Смотрите Выбирают Datastore for File Format или Application для получения дополнительной информации.

Кроме хранилищ данных, MATLAB также имеет несколько других функций, чтобы загрузить части файлов, такие как matfile функционируйте, чтобы загрузить фрагменты MAT-файлов. Эта таблица суммирует частичные функции загрузки типом файла.

FileType Частичная загрузка
Matfile

Загрузите часть переменной путем индексации в объект, который вы создаете с matfile функция. Смотрите Большие данные в Файлах MAT для примера этого использования.

Текст

Используйте textscan функционируйте, чтобы получить доступ к частям файла крупного текста путем чтения только выбранных столбцов и строк. Если вы задаете количество строк или повторный номер формата с textscan, MATLAB вычисляет точный объем памяти, требуемый заранее.

Двоичный файл

Можно использовать низкоуровневые функции ввода-вывода двоичного файла, такие как fread, к частям доступа любого файла, который имеет известный формат. Для двоичных файлов неизвестного формата попытайтесь использовать размещение в ОЗУ с memmapfile функция.

Отобразите, HDF, аудио и видео

Многие функции MATLAB, которые поддерживают загрузку от этих типов файлов, позволяют вам выбирать фрагменты данных, чтобы читать. Для получения дополнительной информации смотрите страницы ссылки на функцию, перечисленные в Поддерживаемых Форматах файлов для Импорта и экспорта.

Увеличение системной области подкачки

Общая память, доступная для приложений на вашем компьютере, состоит из физической памяти (RAM), плюс page file или swap file, на диске. Файл подкачки может быть очень большим (например, 512 терабайт на 64-битном Windows®). Операционная система выделяет виртуальную память для каждого процесса к физической памяти или к файлу подкачки, в зависимости от потребностей системы и других процессов. Увеличение размера файла подкачки может увеличить общую доступную память, но также и обычно приводит к более медленной производительности.

Большинство систем позволяет вам управлять размером своего файла подкачки. Включенные шаги зависят от вашей операционной системы:

  • Windows Systems Используйте Панель управления Windows Control Panel, чтобы изменить размер страничного файла виртуальной памяти в вашей системе. Для получения дополнительной информации обратитесь к справке Windows.

  • Linux® Systems — Измените свою область подкачки при помощи mkswap и swapon команды. Для получения дополнительной информации, в подсказке Linux вводят man сопровождаемый названием команды.

Нет никакого интерфейса для того, чтобы непосредственно управлять областью подкачки в macOS системах.

Установление предела процесса для систем Linux

process limit является максимальной суммой виртуальной памяти, к которой может обратиться один процесс (или приложение). В маловероятном случае вы установили эту настройку, это должно быть достаточно большим, чтобы разместить:

  • Все данные к процессу

  • Файлы программы MATLAB

  • Сам исполняемый файл MATLAB

  • Дополнительная информация состояния

64-битные операционные системы поддерживают предел процесса 8 терабайт. В системах Linux смотрите ulimit команда, чтобы просмотреть и пределы пользователя аппарата включая виртуальную память.

Отключение Java VM в системах Linux

В системах Linux, если вы запускаете MATLAB без Java® JVM™, можно увеличить доступную память рабочей области приблизительно на 400 мегабайтов. Чтобы запустить MATLAB без JVM Java, используйте параметр командной строки -nojvm. Эта опция также увеличивает размер самого большого непрерывного блока памяти приблизительно той же суммой. Путем увеличения самого большого непрерывного блока памяти вы увеличиваете самый большой матричный размер.

Используя -nojvm идет со штрафом в этом, вы теряете много функций, которые используют программное обеспечение Java, включая целую среду разработки. Стартовый MATLAB с -nodesktop опция не сохраняет значительного количества памяти.

Смотрите также

memory

Похожие темы

  • Стратегии эффективного использования памяти
  • Большие файлы и Большие данные
  • Работа с удаленными данными
  • Настройки Java Heap Memory

Issue

When your code operates on large amounts of data or does not use memory
efficiently, MATLAB® might produce an error in response to an unreasonable array size, or
it might run out of memory. MATLAB has built-in protection against creating arrays that are too large.
For example, this code results in an error, because MATLAB cannot create an array with the requested number of elements.

Requested array exceeds the maximum possible variable size.

By default, MATLAB can use up to 100% of the RAM (not including virtual memory) of your
computer to allocate memory for arrays, and if an array size would exceed that
threshold, then MATLAB produces an error. For example, this code attempts to create an array
whose size exceeds the maximum array size limit.

Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size preference (63.7GB). This might cause MATLAB to become
unresponsive.

If you turn off the array size limit in MATLAB Workspace
Preferences
, attempting to create an unreasonably large array might
cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory paging
(that is, moving memory pages between RAM and disk).

Possible Solutions

No matter how you run into memory limits, MATLAB provides several solutions depending on your situation and goals. For
example, you can improve the way your code uses memory, leverage specialized data
structures such as datastores and tall arrays, take advantage of pooled resources
within a computing cluster, or make adjustments to your settings and
preferences.

Note

The solutions presented here are specific to MATLAB. To optimize system-wide memory performance, consider adding more
physical memory (RAM) to your computer or making adjustments at the operating
system level.

Clear Variables When No Longer Needed

Make it a practice to clear variables when they are no longer needed. To clear
items from memory, use the clear function.

Before After
A = rand(1e4);
disp(max(A,[],"all"))
B = rand(1e4);
A = rand(1e4);
disp(max(A,[],"all"))
clear A
B = rand(1e4);

For more information, see Strategies for Efficient Use of Memory.

Use Appropriate Data Storage

Memory requirements differ for MATLAB data types. You might be able to reduce the amount of memory used
by your code by using the appropriate data type and storage. For more
information about the solutions in this section, see Strategies for Efficient Use of Memory.

Use the Appropriate Numeric Class.  The numeric class you should use depends on your intended actions. In
MATLAB, double is the default numeric data type
and provides sufficient precision for most computational tasks:

  • If you want to perform complicated math such as linear algebra,
    use floating-point numbers in either double-precision
    (double) or single-precision
    (single) format. Numbers of type
    single require less memory than numbers of
    type double, but are also represented to less
    precision.

  • If you just need to carry out simple arithmetic and you represent
    the original data as integers, use the integer classes in
    MATLAB.

Class (Data Type) Bytes Supported Operations
single 4 Most math
double 8 All math
logical 1 Logical/conditional operations
int8, uint8 1 Arithmetic and some simple functions
int16, uint16 2 Arithmetic and some simple functions
int32, uint32 4 Arithmetic and some simple functions
int64, uint64 8 Arithmetic and some simple functions

Reduce the Amount of Overhead When Storing Data.  When you create a numeric or character array, MATLAB allocates a block of memory to store the array data.
MATLAB also stores information about the array data, such as its
class and dimensions, in a small, separate block of memory called a
header. Because simple numeric and character arrays
have the least overhead, use them whenever possible. Use other data
structures only for data that is too complex to store in a simple
array.

For structures and cell arrays, MATLAB creates a header not only for the array, but also for each
field of the structure or each cell of the cell array. Therefore, the amount
of memory required to store a structure or cell array depends not only on
how much data it holds, but also on how it is constructed. As a result, cell
arrays with many small elements or structures with many fields containing
little content have large overhead and should be avoided.

Before After
% S has 15,000 fields (3 fields per array element)
for i = 1:100
    for j = 1:50
        S(i,j).R = 0;  % Each field contains a numeric scalar
        S(i,j).G = 0;
        S(i,j).B = 0;
    end
end
% S has 3 fields 
S.R = zeros(100,50);  % Each field contains a numeric array
S.G = zeros(100,50);
S.B = zeros(100,50);

Make Arrays Sparse When Possible.  A good practice is to store matrices with few nonzero elements using
sparse storage. When a full matrix has a small number of nonzero elements,
converting the matrix to sparse storage typically improves memory usage and
code execution time. MATLAB has several functions that support sparse storage. For
example, you can use the speye function to create a
sparse identity matrix.

Before After
I = eye(1000);
I = speye(1000);

Import Data Using the Appropriate MATLAB Class.  When reading data from a binary file with fread, a common mistake is
to specify only the class of the data in the file and not the class of the
data MATLAB uses once it is in the workspace. If you do not specify the
class of the data in memory, MATLAB uses double even if you read 8-bit
values.

Before After
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8"); 
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8=>uint8"); 

Avoid Unnecessary Copies of Data

To improve memory usage and execution speed, make sure your code does not
result in unnecessary copies of data. For more information about the solutions
in this section, see Avoid Unnecessary Copies of Data and Strategies for Efficient Use of Memory.

Avoid Creating Temporary Arrays.  Avoid creating temporary arrays when they are not necessary. For example,
instead of creating an array of zeros as a temporary variable and then
passing that variable to a function, use one command to do both
operations.

Before After
A = zeros(1e6,1);
As = single(A);
As = zeros(1e6,1,"single");

Preallocate Memory.  When you work with large data sets, repeatedly resizing arrays might cause
your program to run out of memory. If you expand an array beyond the
available contiguous memory of its original location, MATLAB must make a copy of the array and move the copy into a memory
block with sufficient space. During this process, two copies of the original
array exist in memory. You can improve the memory usage and code execution
time by preallocating the maximum amount of space required for the array.
For more information, see Preallocation.

Before After
x = 0;
for k = 2:1000000
   x(k) = x(k-1) + 5;
end
x = zeros(1,1000000);
for k = 2:1000000
   x(k) = x(k-1) + 5;
end

Use Nested Functions to Pass Fewer Arguments.  When calling a function, MATLAB typically makes a temporary copy of the variable in the
caller’s workspace if the function modifies its value. MATLAB applies various techniques to avoid making unnecessary copies,
but avoiding a temporary copy of an input variable is not always
possible.

One way to avoid temporary copies in function calls is to use nested
functions. A nested function shares the workspace of all outer functions, so
you do not need to pass a copy of variables in the function call. For more
information, see Nested Functions.

Load Only as Much Data as You Need

One way to fix memory issues is to import into MATLAB only as much of a large data set as you need for the problem you
are trying to solve. Data set size is not usually a problem when importing from
sources such as a database, where you can explicitly search for elements
matching a query. But it is a common problem with loading large flat text or
binary files.

The datastore function lets you
work with large data sets incrementally. Datastores are useful any time you want
to load only some portions of a data set into memory at a time.

To create a datastore, provide the name of a file or a directory containing a
collection of files with similar formatting. For example, with a single file,
use the
following.

ds = datastore("path/to/file.csv");

Or
with a collection of files in a folder, use the
following.

ds = datastore("path/to/folder/");

You
also can use the wildcard character * to select all files of
a specific
type.

ds = datastore("path/to/*.csv");

Datastores
support a wide variety of file formats (tabular data, images, spreadsheets, and
so on). For more information, see Select Datastore for File Format or Application.

Aside from datastores, MATLAB also has several other functions to load parts of files. This
table summarizes the functions by the type of files they operate on.

File Type Partial Loading
MAT-file

Load part of a variable by indexing into an object that
you create with the matfile
function. For more information, see Save and Load Parts of Variables in MAT-Files.

Text

Use the textscan
function to access parts of a large text file by reading
only the selected columns and rows. If you specify the
number of rows or a repeat format number with
textscan, MATLAB calculates the exact amount of memory required
beforehand.

Binary

You can use low-level binary file I/O functions, such
as fread, to
access parts of any file that has a known format. For binary
files of an unknown format, try using memory mapping with
the memmapfile
function.

Image, Audio, Video, and HDF

Many of the MATLAB functions that support loading from these
types of files allow you to select portions of the data to
read. For details, see the function reference pages listed
in Supported File Formats for Import and Export.

Use Tall Arrays

Tall arrays help you work with data sets that are too large to fit in memory.
MATLAB works with small blocks of the data at a time, automatically
handling all of the data chunking and processing in the background. You can use
tall arrays in two primary ways:

  • If you have a large array that fits in memory, but you run out of
    memory when you try to perform calculations, you can cast the array
    to a tall
    array.

    This
    approach lets you work with large arrays that can fit in memory, but
    that consume too much memory to allow for copies of the data during
    calculations. For example, if you have 8 GB of RAM and a 5 GB
    matrix, casting the matrix to a tall array enables you to perform
    calculations on the matrix without running out of memory. For an
    example of this usage, see tall.

  • If you have file- or folder-based data, you can create a datastore
    and then create a tall array on top of the
    datastore.

    ds = datastore("path/to/file.csv");
    t = tall(ds);

    This
    approach gives you the full power of tall arrays in MATLAB. The data can have any number of rows and MATLAB does not run out of memory. And because
    datastore works with both local and remote
    data locations, the data you work with does not need to be on the
    computer you use to analyze it. See Work with Remote Data for more
    information.

To learn more about tall arrays, see Tall Arrays for Out-of-Memory Data.

Use Memory of Multiple Machines

If you have a cluster of computers, you can use distributed arrays (requires
Parallel Computing Toolbox™) to perform calculations using the combined memory of all the
machines in the cluster. Depending on how your data fits in memory, different
ways to partition data among workers of a parallel pool exist. For more
information, see Distributing Arrays to Parallel Workers (Parallel Computing Toolbox).

Adjust Settings and Preferences

Generally, rewriting code is the most effective way to improve memory
performance. However, if you cannot make changes to your code, these solutions
might provide it with the required amount of memory.

Start MATLAB Without Java Virtual Machine or Decrease the Java Heap Size.  If you either start MATLAB without the Java® Virtual Machine (JVM™) software or decrease the Java heap size, you can increase the available workspace memory. To
start MATLAB without JVM, use the command-line option -nojvm. For
information on how to decrease the Java heap size, see Java Heap Memory Preferences.

Using -nojvm comes with a penalty in that you lose
several features that rely on JVM, such as the desktop tools and graphics. Starting MATLAB with the -nodesktop option does not save
any substantial amount of memory.

Adjust the Array Size Limit.  If you face an error stating that the array size exceeds the maximum array
size preference, you can adjust this array size limit in MATLAB. For information on adjusting the array size limit, see Workspace and Variable Preferences. This solution is helpful only if the
array you want to create exceeds the current maximum array size limit but is
not too large to fit in memory. Even if you turn off the array size limit,
attempting to create an unreasonably large array might cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory
paging.

See Also

memory | whos | datastore | tall

Related Topics

  • How MATLAB Allocates Memory
  • Strategies for Efficient Use of Memory
  • Avoid Unnecessary Copies of Data
  • Large Files and Big Data

When I run a sample script in MATLAB, it says:

Out of memory. Type HELP MEMORY for your options.

When I type «memory», it reports:

Maximum possible array:             156 MB (1.638e+008 bytes) *
Memory available for all arrays:    740 MB (7.756e+008 bytes) **
Memory used by MATLAB:             1054 MB (1.105e+009 bytes)
Physical Memory (RAM):             3070 MB (3.219e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

Is there any way to get around this error? I’m using Windows XP x32 with MATLAB 2009a.

Peter Mortensen's user avatar

asked Aug 14, 2009 at 11:08

Contango's user avatar

2

pack does a memory defragmentation. It might help you a bit as far as the contiguous memory available.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 11:39

Marcin's user avatar

MarcinMarcin

3,3801 gold badge22 silver badges15 bronze badges

0

Remember, when MATLAB says it’s out of memory, it means it’s out of contiguous memory, so rebooting or restarting MATLAB may work.

But, I’d recommend optimizing your code and identifying how you’re eating up so much memory. It could be an ill-designed recursive loop, or a bad indexing function (using doubles instead of logicals to index a huge matrix).

I practically lived with memory errors for a while since I was dealing with huge datasets, but there’s always a workaround, ask specific questions and you’ll be surprised.

Community's user avatar

answered Aug 14, 2009 at 12:19

Jacob's user avatar

JacobJacob

34.1k14 gold badges109 silver badges165 bronze badges

Problem fixed.

Under Windows XP x32, I managed to almost double the amount of memory available to MATLAB by editing boot.ini to add the switch /3GB /USERVA=3030

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB /USERVA=3030

Together with reducing our array sizes, this completely fixed the problem :)

I could have also fixed the problem by upgrading to Windows x64 or Windows 7 x64. This act also doubles the amount of memory available to MATLAB, even if you stick with MATLAB x32 and don’t upgrade to MATLAB x64. Windows x64 is just far more memory efficient, even with systems that only have 4 GB of physical RAM installed.

Peter Mortensen's user avatar

answered Aug 17, 2009 at 11:08

Contango's user avatar

ContangoContango

74.4k57 gold badges251 silver badges300 bronze badges

5

Try this, it works well for me.

  • Go to Home -> Preference icon -> General -> Java Heap Memory -> Allocate what size of memory you want
  • In Preference window, go to «Workspace» (out of Java heap memory level) -> See «Matlab Array size limit»
    Make sure uncheck the ‘Limit the maximum array size to a percentage of RAM’. Because you want to extend memory
    so we don’t need this feature.
  • Done.

answered Mar 22, 2017 at 16:04

Vinh Trieu's user avatar

Vinh TrieuVinh Trieu

9638 silver badges12 bronze badges

What are you attempting to allocate when it runs out of memory (OOM)? Do you have code to reproduce? A wide range of problems can cause out of memory errors.

To diagnose, use «dbstop if all error» to set a breakpoint on errors. The out of memory will trigger this, and you can use dbup, dbdown, and whos() to see what’s consuming memory. Often an OOM is caused by a bad array size or index calculation, not just by big data structures. E.g. this will trigger an OOM in pretty much any 32-bit MATLAB.

>> x = 1;
>> x(2^30) = 2
??? Out of memory. Type HELP MEMORY for your options.

Peter Mortensen's user avatar

answered Aug 14, 2009 at 16:15

Andrew Janke's user avatar

Andrew JankeAndrew Janke

23.4k5 gold badges55 silver badges85 bronze badges

0

I faced a similar error while running an (old) C file in MATLAB using mex.

I found my solution at this issue on GitLab.

First, uncheck the option «Limit the maximum Array Size to a % of RAM» located under Preferences -> Workspace, as also indicated in this earlier answer.

Once applied, run your C file in the command window using

mex filename.c -compatibleArrayDims

Cris Luengo's user avatar

Cris Luengo

53.1k9 gold badges63 silver badges118 bronze badges

answered Sep 14, 2020 at 10:40

VIREN RAMCHANDANI's user avatar

4

Issue

When your code operates on large amounts of data or does not use memory
efficiently, MATLAB® might produce an error in response to an unreasonable array size, or
it might run out of memory. MATLAB has built-in protection against creating arrays that are too large.
For example, this code results in an error, because MATLAB cannot create an array with the requested number of elements.

Requested array exceeds the maximum possible variable size.

By default, MATLAB can use up to 100% of the RAM (not including virtual memory) of your
computer to allocate memory for arrays, and if an array size would exceed that
threshold, then MATLAB produces an error. For example, this code attempts to create an array
whose size exceeds the maximum array size limit.

Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size preference (63.7GB). This might cause MATLAB to become
unresponsive.

If you turn off the array size limit in MATLAB Workspace
Preferences
, attempting to create an unreasonably large array might
cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory paging
(that is, moving memory pages between RAM and disk).

Possible Solutions

No matter how you run into memory limits, MATLAB provides several solutions depending on your situation and goals. For
example, you can improve the way your code uses memory, leverage specialized data
structures such as datastores and tall arrays, take advantage of pooled resources
within a computing cluster, or make adjustments to your settings and
preferences.

Note

The solutions presented here are specific to MATLAB. To optimize system-wide memory performance, consider adding more
physical memory (RAM) to your computer or making adjustments at the operating
system level.

Clear Variables When No Longer Needed

Make it a practice to clear variables when they are no longer needed. To clear
items from memory, use the clear function.

Before After
A = rand(1e4);
disp(max(A,[],"all"))
B = rand(1e4);
A = rand(1e4);
disp(max(A,[],"all"))
clear A
B = rand(1e4);

For more information, see Strategies for Efficient Use of Memory.

Use Appropriate Data Storage

Memory requirements differ for MATLAB data types. You might be able to reduce the amount of memory used
by your code by using the appropriate data type and storage. For more
information about the solutions in this section, see Strategies for Efficient Use of Memory.

Use the Appropriate Numeric Class.  The numeric class you should use depends on your intended actions. In
MATLAB, double is the default numeric data type
and provides sufficient precision for most computational tasks:

  • If you want to perform complicated math such as linear algebra,
    use floating-point numbers in either double-precision
    (double) or single-precision
    (single) format. Numbers of type
    single require less memory than numbers of
    type double, but are also represented to less
    precision.

  • If you just need to carry out simple arithmetic and you represent
    the original data as integers, use the integer classes in
    MATLAB.

Class (Data Type) Bytes Supported Operations
single 4 Most math
double 8 All math
logical 1 Logical/conditional operations
int8, uint8 1 Arithmetic and some simple functions
int16, uint16 2 Arithmetic and some simple functions
int32, uint32 4 Arithmetic and some simple functions
int64, uint64 8 Arithmetic and some simple functions

Reduce the Amount of Overhead When Storing Data.  When you create a numeric or character array, MATLAB allocates a block of memory to store the array data.
MATLAB also stores information about the array data, such as its
class and dimensions, in a small, separate block of memory called a
header. Because simple numeric and character arrays
have the least overhead, use them whenever possible. Use other data
structures only for data that is too complex to store in a simple
array.

For structures and cell arrays, MATLAB creates a header not only for the array, but also for each
field of the structure or each cell of the cell array. Therefore, the amount
of memory required to store a structure or cell array depends not only on
how much data it holds, but also on how it is constructed. As a result, cell
arrays with many small elements or structures with many fields containing
little content have large overhead and should be avoided.

Before After
% S has 15,000 fields (3 fields per array element)
for i = 1:100
    for j = 1:50
        S(i,j).R = 0;  % Each field contains a numeric scalar
        S(i,j).G = 0;
        S(i,j).B = 0;
    end
end
% S has 3 fields 
S.R = zeros(100,50);  % Each field contains a numeric array
S.G = zeros(100,50);
S.B = zeros(100,50);

Make Arrays Sparse When Possible.  A good practice is to store matrices with few nonzero elements using
sparse storage. When a full matrix has a small number of nonzero elements,
converting the matrix to sparse storage typically improves memory usage and
code execution time. MATLAB has several functions that support sparse storage. For
example, you can use the speye function to create a
sparse identity matrix.

Before After
I = eye(1000);
I = speye(1000);

Import Data Using the Appropriate MATLAB Class.  When reading data from a binary file with fread, a common mistake is
to specify only the class of the data in the file and not the class of the
data MATLAB uses once it is in the workspace. If you do not specify the
class of the data in memory, MATLAB uses double even if you read 8-bit
values.

Before After
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8"); 
fileID = fopen("large_file_of_uint8s.bin","r"); 
A = fread(fileID,1e3,"uint8=>uint8"); 

Avoid Unnecessary Copies of Data

To improve memory usage and execution speed, make sure your code does not
result in unnecessary copies of data. For more information about the solutions
in this section, see Avoid Unnecessary Copies of Data and Strategies for Efficient Use of Memory.

Avoid Creating Temporary Arrays.  Avoid creating temporary arrays when they are not necessary. For example,
instead of creating an array of zeros as a temporary variable and then
passing that variable to a function, use one command to do both
operations.

Before After
A = zeros(1e6,1);
As = single(A);
As = zeros(1e6,1,"single");

Preallocate Memory.  When you work with large data sets, repeatedly resizing arrays might cause
your program to run out of memory. If you expand an array beyond the
available contiguous memory of its original location, MATLAB must make a copy of the array and move the copy into a memory
block with sufficient space. During this process, two copies of the original
array exist in memory. You can improve the memory usage and code execution
time by preallocating the maximum amount of space required for the array.
For more information, see Preallocation.

Before After
x = 0;
for k = 2:1000000
   x(k) = x(k-1) + 5;
end
x = zeros(1,1000000);
for k = 2:1000000
   x(k) = x(k-1) + 5;
end

Use Nested Functions to Pass Fewer Arguments.  When calling a function, MATLAB typically makes a temporary copy of the variable in the
caller’s workspace if the function modifies its value. MATLAB applies various techniques to avoid making unnecessary copies,
but avoiding a temporary copy of an input variable is not always
possible.

One way to avoid temporary copies in function calls is to use nested
functions. A nested function shares the workspace of all outer functions, so
you do not need to pass a copy of variables in the function call. For more
information, see Nested Functions.

Load Only as Much Data as You Need

One way to fix memory issues is to import into MATLAB only as much of a large data set as you need for the problem you
are trying to solve. Data set size is not usually a problem when importing from
sources such as a database, where you can explicitly search for elements
matching a query. But it is a common problem with loading large flat text or
binary files.

The datastore function lets you
work with large data sets incrementally. Datastores are useful any time you want
to load only some portions of a data set into memory at a time.

To create a datastore, provide the name of a file or a directory containing a
collection of files with similar formatting. For example, with a single file,
use the
following.

ds = datastore("path/to/file.csv");

Or
with a collection of files in a folder, use the
following.

ds = datastore("path/to/folder/");

You
also can use the wildcard character * to select all files of
a specific
type.

ds = datastore("path/to/*.csv");

Datastores
support a wide variety of file formats (tabular data, images, spreadsheets, and
so on). For more information, see Select Datastore for File Format or Application.

Aside from datastores, MATLAB also has several other functions to load parts of files. This
table summarizes the functions by the type of files they operate on.

File Type Partial Loading
MAT-file

Load part of a variable by indexing into an object that
you create with the matfile
function. For more information, see Save and Load Parts of Variables in MAT-Files.

Text

Use the textscan
function to access parts of a large text file by reading
only the selected columns and rows. If you specify the
number of rows or a repeat format number with
textscan, MATLAB calculates the exact amount of memory required
beforehand.

Binary

You can use low-level binary file I/O functions, such
as fread, to
access parts of any file that has a known format. For binary
files of an unknown format, try using memory mapping with
the memmapfile
function.

Image, Audio, Video, and HDF

Many of the MATLAB functions that support loading from these
types of files allow you to select portions of the data to
read. For details, see the function reference pages listed
in Supported File Formats for Import and Export.

Use Tall Arrays

Tall arrays help you work with data sets that are too large to fit in memory.
MATLAB works with small blocks of the data at a time, automatically
handling all of the data chunking and processing in the background. You can use
tall arrays in two primary ways:

  • If you have a large array that fits in memory, but you run out of
    memory when you try to perform calculations, you can cast the array
    to a tall
    array.

    This
    approach lets you work with large arrays that can fit in memory, but
    that consume too much memory to allow for copies of the data during
    calculations. For example, if you have 8 GB of RAM and a 5 GB
    matrix, casting the matrix to a tall array enables you to perform
    calculations on the matrix without running out of memory. For an
    example of this usage, see tall.

  • If you have file- or folder-based data, you can create a datastore
    and then create a tall array on top of the
    datastore.

    ds = datastore("path/to/file.csv");
    t = tall(ds);

    This
    approach gives you the full power of tall arrays in MATLAB. The data can have any number of rows and MATLAB does not run out of memory. And because
    datastore works with both local and remote
    data locations, the data you work with does not need to be on the
    computer you use to analyze it. See Work with Remote Data for more
    information.

To learn more about tall arrays, see Tall Arrays for Out-of-Memory Data.

Use Memory of Multiple Machines

If you have a cluster of computers, you can use distributed arrays (requires
Parallel Computing Toolbox™) to perform calculations using the combined memory of all the
machines in the cluster. Depending on how your data fits in memory, different
ways to partition data among workers of a parallel pool exist. For more
information, see Distributing Arrays to Parallel Workers (Parallel Computing Toolbox).

Adjust Settings and Preferences

Generally, rewriting code is the most effective way to improve memory
performance. However, if you cannot make changes to your code, these solutions
might provide it with the required amount of memory.

Start MATLAB Without Java Virtual Machine or Decrease the Java Heap Size.  If you either start MATLAB without the Java® Virtual Machine (JVM™) software or decrease the Java heap size, you can increase the available workspace memory. To
start MATLAB without JVM, use the command-line option -nojvm. For
information on how to decrease the Java heap size, see Java Heap Memory Preferences.

Using -nojvm comes with a penalty in that you lose
several features that rely on JVM, such as the desktop tools and graphics. Starting MATLAB with the -nodesktop option does not save
any substantial amount of memory.

Adjust the Array Size Limit.  If you face an error stating that the array size exceeds the maximum array
size preference, you can adjust this array size limit in MATLAB. For information on adjusting the array size limit, see Workspace and Variable Preferences. This solution is helpful only if the
array you want to create exceeds the current maximum array size limit but is
not too large to fit in memory. Even if you turn off the array size limit,
attempting to create an unreasonably large array might cause MATLAB to run out of memory, or it might make MATLAB or even your computer unresponsive due to excessive memory
paging.

See Also

memory | whos | datastore | tall

Related Topics

  • How MATLAB Allocates Memory
  • Strategies for Efficient Use of Memory
  • Avoid Unnecessary Copies of Data
  • Large Files and Big Data

Решение “Из памяти” ошибок

Эта тема объясняет несколько стратегий, которые можно использовать в ситуациях, где MATLAB® исчерпывает память. MATLAB является 64-битным приложением, которое работает на 64-битных операционных системах. Это возвращает сообщение об ошибке каждый раз, когда это запрашивает сегмент памяти от операционной системы, которая больше, чем, что доступно.

MATLAB имеет встроенную защиту от создания массивов, которые являются слишком большими. По умолчанию MATLAB может использовать до 100% RAM (не включая виртуальную память) вашего компьютера, чтобы выделить память для массивов, и если массив превысил бы тот порог, то MATLAB возвращает ошибку. Например, этот оператор пытается создать массив с неблагоразумным размером:

Error using rand
Requested 1000000x1000000 (7450.6GB) array exceeds maximum array size
preference. Creation of arrays greater than this limit may take a long
time and cause MATLAB to become unresponsive. See array size limit or
preference panel for more information.

Смотрите Рабочую область и Настройки переменной для получения информации о корректировке этого предела размера массивов. Если вы выключаете предел размера массивов, то MATLAB возвращает различную ошибку:

Out of memory. Type "help memory" for your options.

Неважно, как вы сталкиваетесь с пределами памяти, существует несколько разрешений, доступных в зависимости от ваших целей. Методы, обсужденные в Стратегиях Эффективного использования Памяти, могут помочь вам оптимизировать доступную память, включая которую вы имеете:

  • Используйте соответствующее хранение данных

  • Предотвращение создавать Временные массивы

  • Исправьте используемую память

Если вы уже используете память эффективно, и проблема сохраняется, то остающиеся разделы этой страницы содержат возможные решения.

Усильте tall Массивы

Длинные массивы для Данных, которые не помещаются в память, спроектированы, чтобы помочь вам работать с наборами данных, которые являются слишком большими, чтобы поместиться в память. MATLAB работает с маленькими блоками данных за один раз, автоматически обрабатывая все разделение на блоки данных и обработку в фоновом режиме. Существует два первичных способа, которыми можно усилить длинные массивы:

  1. Если у вас есть большой массив, который умещается в памяти, но у вас заканчивается память, когда вы пытаетесь выполнить вычисления, можно бросить массив к длинному массиву:

    Этот метод включает вам, работают с большими массивами, которые могут уместиться в памяти, но которые используют слишком много памяти, чтобы допускать копии данных во время вычислений. Например, если у вас есть 8 ГБ RAM, и матрица на 5 ГБ, бросая матрицу к длинному массиву позволяет вам выполнить вычисления на матрице, не исчерпывая память. Смотрите Преобразуют Массивы В оперативной памяти в Длинные массивы для примера этого использования.

  2. Если у вас есть файл или основанные на папке данные, можно создать datastore и затем создайте высокий массив поверх datastore:

    ds = datastore('path/to/data.csv');
    tt = tall(ds);

    Этот метод дает вам полную мощность длинных массивов в MATLAB: данные могут иметь любое количество строк, и в MATLAB не заканчивается память. И потому что datastore работает и с районами локальных и с удаленных данных, данные, с которыми вы работаете, не должны быть на компьютере, который вы используете, чтобы анализировать их. Смотрите работу с Удаленными данными для получения дополнительной информации.

Усильте память о нескольких машинах

Если у вас есть кластер компьютеров, можно использовать Parallel Computing Toolbox™ и Распределенные Массивы (Parallel Computing Toolbox), чтобы выполнить вычисления с помощью объединенной памяти обо всех машинах в кластере. Это позволяет вам работать с целым распределенным массивом как одна сущность. Однако рабочие действуют только с их стороны массива, и автоматически передают данные между собой при необходимости.

Создание распределенного массива очень похоже на создание длинного массива:

ds = datastore('path/to/data.csv');
dt = distributed(ds);

Загрузите только столько данные, сколько вам нужно

Другой возможный способ зафиксировать проблемы памяти состоит в том, чтобы только импортировать в MATLAB такое количество большого набора данных, как вам нужно для задачи, которую вы пытаетесь решить. Это обычно не проблема при импорте из источников, таких как база данных, где можно явным образом искать элементы, совпадающие с запросом. Но это — типичная проблема с загрузкой большого плоского текста или двоичных файлов.

datastore функция позволяет вам работать с большими наборами данных инкрементно. Эта функция подкрепляет Длинные массивы для Данных, которые не помещаются в память, и Распределенных Массивов (Parallel Computing Toolbox), но можно использовать его для других целей также. Хранилища данных полезны любое время, вы хотите загрузить небольшие части набора данных в память за один раз.

Чтобы создать datastore, необходимо обеспечить имя файла или директории, содержащей набор файлов с подобным форматированием. Например, с одним файлом:

ds = datastore('path/to/file.csv')

Или, с набором файлов в папке:

ds = datastore('path/to/folder/')

Можно также использовать подстановочный символ * выбрать все файлы определенного типа, как в:

ds = datastore('data/*.csv')

Хранилища данных поддерживают большое разнообразие форматов общего файла (табличные данные, изображения, электронные таблицы, и так далее). Смотрите Выбирают Datastore for File Format или Application для получения дополнительной информации.

Кроме хранилищ данных, MATLAB также имеет несколько других функций, чтобы загрузить части файлов, такие как matfile функционируйте, чтобы загрузить фрагменты MAT-файлов. Эта таблица суммирует частичные функции загрузки типом файла.

FileType Частичная загрузка
Matfile

Загрузите часть переменной путем индексации в объект, который вы создаете с matfile функция. Смотрите Большие данные в Файлах MAT для примера этого использования.

Текст

Используйте textscan функционируйте, чтобы получить доступ к частям файла крупного текста путем чтения только выбранных столбцов и строк. Если вы задаете количество строк или повторный номер формата с textscan, MATLAB вычисляет точный объем памяти, требуемый заранее.

Двоичный файл

Можно использовать низкоуровневые функции ввода-вывода двоичного файла, такие как fread, к частям доступа любого файла, который имеет известный формат. Для двоичных файлов неизвестного формата попытайтесь использовать размещение в ОЗУ с memmapfile функция.

Отобразите, HDF, аудио и видео

Многие функции MATLAB, которые поддерживают загрузку от этих типов файлов, позволяют вам выбирать фрагменты данных, чтобы читать. Для получения дополнительной информации смотрите страницы ссылки на функцию, перечисленные в Поддерживаемых Форматах файлов для Импорта и экспорта.

Увеличение системной области подкачки

Общая память, доступная для приложений на вашем компьютере, состоит из физической памяти (RAM), плюс page file или swap file, на диске. Файл подкачки может быть очень большим (например, 512 терабайт на 64-битном Windows®). Операционная система выделяет виртуальную память для каждого процесса к физической памяти или к файлу подкачки, в зависимости от потребностей системы и других процессов. Увеличение размера файла подкачки может увеличить общую доступную память, но также и обычно приводит к более медленной производительности.

Большинство систем позволяет вам управлять размером своего файла подкачки. Включенные шаги зависят от вашей операционной системы:

  • Windows Systems Используйте Панель управления Windows Control Panel, чтобы изменить размер страничного файла виртуальной памяти в вашей системе. Для получения дополнительной информации обратитесь к справке Windows.

  • Linux® Systems — Измените свою область подкачки при помощи mkswap и swapon команды. Для получения дополнительной информации, в подсказке Linux вводят man сопровождаемый названием команды.

Нет никакого интерфейса для того, чтобы непосредственно управлять областью подкачки в macOS системах.

Установление предела процесса для систем Linux

process limit является максимальной суммой виртуальной памяти, к которой может обратиться один процесс (или приложение). В маловероятном случае вы установили эту настройку, это должно быть достаточно большим, чтобы разместить:

  • Все данные к процессу

  • Файлы программы MATLAB

  • Сам исполняемый файл MATLAB

  • Дополнительная информация состояния

64-битные операционные системы поддерживают предел процесса 8 терабайт. В системах Linux смотрите ulimit команда, чтобы просмотреть и пределы пользователя аппарата включая виртуальную память.

Отключение Java VM в системах Linux

В системах Linux, если вы запускаете MATLAB без Java® JVM™, можно увеличить доступную память рабочей области приблизительно на 400 мегабайтов. Чтобы запустить MATLAB без JVM Java, используйте параметр командной строки -nojvm. Эта опция также увеличивает размер самого большого непрерывного блока памяти приблизительно той же суммой. Путем увеличения самого большого непрерывного блока памяти вы увеличиваете самый большой матричный размер.

Используя -nojvm идет со штрафом в этом, вы теряете много функций, которые используют программное обеспечение Java, включая целую среду разработки. Стартовый MATLAB с -nodesktop опция не сохраняет значительного количества памяти.

Смотрите также

memory

Похожие темы

  • Стратегии эффективного использования памяти
  • Большие файлы и Большие данные
  • Работа с удаленными данными
  • Настройки Java Heap Memory

«Out of Memory» Errors

Typically, MATLAB generates an Out of Memory message whenever it requests a segment of memory from the operating system that is larger than what is currently available. When you see this message, use any of the techniques discussed earlier in this section to help optimize the available memory.

If the Out of Memory message still appears, you can try any of the following:

  • Increase the size of the swap file. We recommend that your machine be configured with twice as much swap space as you have RAM.
  • See «Increasing the System Swap Space», below.
  • Allocate larger matrices earlier in the MATLAB session
  • If possible, break large matrices into several smaller matrices so that less memory is used at any one time.
  • Reduce the size of your data.
  • Make sure that there are no external constraints on the memory accessible to MATLAB. (On UNIX systems, use the limit command to check).
  • On UNIX systems, you can run memory intensive tasks with more available address space by starting MATLAB with the -nojvm switch.
  • See Running MATLAB Without the Java Virtual Machine.
  • Add more memory to the system.

Increasing the System Swap Space

How you set the swap space for your computer depends on what operating system you are running on.

UNIX.   Information about swap space can be procured by typing pstat -s at the UNIX command prompt. For detailed information on changing swap space, ask your system administrator.

Linux.   Swap space can be changed by using the mkswap and swapon commands. For more information on the above commands, type man command_name at the Linux prompt.

Windows 98, Windows NT, and Windows ME.   Follow the steps shown here:

  1. Right-click on the My Computer icon, and select Properties.
  2. Select the Performance tab and click the Change button to change the amount of virtual memory.

Windows 2000.   Follow the steps shown here:

  1. Right-click on the My Computer icon, and select Properties.
  2. Select the Advanced tab and choose Performance Options.
  3. Click on the Change button to change the amount of virtual memory.

Running MATLAB Without the Java Virtual Machine

You are more likely to encounter Out of Memory errors when using MATLAB Version 6.0 or greater due to the use of the Java Virtual Machine (JVM) within MATLAB. The JVM and MATLAB use the same virtual address space, leaving less memory available for your MATLAB applications to use.

MATLAB uses the JVM to implement its GUI-based development environment. You are encouraged to use this environment for your application development work. However, if you get Out of Memory errors while running memory intensive applications, and you are running MATLAB on a UNIX platform, you may want to run these without the JVM.

On UNIX systems, it is possible to run MATLAB without the JVM by starting MATLAB with a -nojvm switch. Type the following at the UNIX prompt:

  • matlab -nojvm
    

When you use this option, you cannot use the desktop or any of the MATLAB tools that require Java.

   Ways to Conserve Memory   Platform-Specific Memory Topics 

В Matlab есть одна неприятная штука — Out of memory error. Фактор управления памятью приходится учитывать при операциях с большими матрицами, которые требуют немпрерывных массивов памяти.
Существует ряд способов решения проблем, связанных с выделением памяти.
Читать про них:
Memory Management Guide
Так, сегодня проблему с очень большой матрицей вектора перемещений в методе конечных элементов UX(i,j) размером 836×2007 мне удалось решить, используя очень простую запись в виде массиыва ячеек UX{i}(j) (массив ячеек хранит указатели на массивы, а не всю матрицу сразу. Просто? А вот я полдня с со своей программой мучился).
Следует отметить, что Matlab вываливался с ошибкой от матрицы, считываемой с диска с явно заданными элементами:

UX(1,1)=5.8;
UX(1,2)=5.4;
UX(1,2)=5.2;
...

Если вы создадите случайную матрицу rand(836,2007) прямо в командной строке, никакой ошибки не случится.

Про массивы ячеек (cell arrays) можно почитать очень дельную статью:
Cell Arrays and their contents

Okay so, I m a student in my final months of a Master’s in Physics. For my research project, my supervisor gave us a bit of code he wrote.

The code reads in about 6 different data points for each of ~300,000 galaxies from a .fits file, and for 2 different aperture sizes, returns a number that tells you the relative density of that galaxy’s surroundings, compared to that of the whole sample.

I work with a partner. We both tried running the same code. The only differences between the code he’s running and the code I’m running are the file paths that we read from/write to.

The partner, on his 6th gen i5 laptop with 8GB RAM, can run the code to completion. It takes him about 4 hours, but it works. I believe he’s on R2019a.

I, with my heavily overclocked i7 6700k and 16GB RAM, get halfway through (the code outputs some info along the way that roughly indicates how much work it has done) before matlab returns an error «Error — Out of memory, type ‘help memory’ for more info». I’m using R2018b.

From looking at some forums quickly last night, I went to Home > preferences > general > java something or other, and given MATLAB all the memory it can ask for. The first half of the code takes about 25 minutes for me to run. So maybe an hour in total compared to my partner’s 4 hours.

MATLAB’s RAM usage with all non-essential processes closed peaks at about 9GB right as MATLAB runs out of memory, it then rests down at about 4GB with all the arrays it has stored.

Why would MATLAB be running out of available memory on my PC, despite having about 5GB left unallocated in my system. Meanwhile my partner doesn’t run out of memory with his 8GB RAM.

I can split it in to two chunks fairly easily, but it’s just bothering me that I can’t make it run as it is when clearly it can run on my system.

  • Ошибка osu was unable to obtain a graphics context
  • Ошибка osloader exe ошибка 0xc0000428 состояние не удается проверить цифровую подпись этого файла
  • Ошибка osf частотный преобразователь
  • Ошибка osd на приставке lumax
  • Ошибка origins ini not found