FileMatrix: File-based matrix which stores sets of columns in separate files. More...

Detailed Description

FileMatrix: File-based matrix which stores sets of columns in separate files.

This class features a caching functionality for the last accessed block, so that for subsequent calls to loadBlock with the same number no new hard drive access is necessary. This makes the FileMatrix almost as fast as a normal matrix when one block is used, i.e. the whole matrix fits into one block of the pre-defined size block_size passed at the constructor.

Author: Daniel Wirtz

Date: 2012-07-09

Change in 0.6:: (Daniel Wirtz, 2012-11-06) Changed the constructor to use an inputParser

New in 0.6:: (Daniel Wirtz, 2012-07-09) Added this class.

This class is part of the framework

KerMor - Model Order Reduction using Kernels

Homepage http://www.morepas.org/software/index.html
Documentation http://www.morepas.org/software/kermor/index.html
License KerMor license conditions

Todo:: add getBlockPos method for block indices and consider removing the idx vector

Definition at line 18 of file FileMatrix.m.

Public Member Functions
	FileMatrix (matrix< double > var, varargin)
	Creates a new file matrix. Possible constructors: More...

function B =	spawnWithContent (matrix< double > A)
	Creates a new FileMatrix containing the matrix A. The matrix is stored at the at the same location as the current matrix and the same block size is used. More...

function [ rowmin , rowmax ] =	getColBoundingBox ()
	Computes the bounding box of the matrix with respect to columns. More...

function pos =	getBlockPos (nr)
	Returns the column indices of the block "nr" within the full matrix. More...

function copy =	copyWithNewBlockSize (block_size)

function	relocate (new_root)
	Relocates this FileMatrix instance to a new folder. More...

function res =	transposedTimes (B)
	Implements the operation A'*B for this matrix A and another FileMatrix B. More...

function s =	sum (dim)
	% Overloaded methods More...

function value =	power (expo)

function n =	numel ()

function [ n , m ] =	size (dim)
	Implementation of ABlockedData.size. More...

function varargout =	subsref (key)
	Implements subscripted value retrieval. More...

function this =	subsasgn (key, value)
	Implements subscripted assignment. More...

function diff =	minus (B)

function a =	mldivide (R)

function AB =	mtimes (B)
	Override of ABlockedData.mtimes. More...

function AB =	times (B)
	Override of ABlockedData.mtimes. More...

function trans =	ctranspose ()

function res =	eq (B)

function res =	ne (B)

function	horzcat (varargin)

function	vertcat (varargin)

function	delete ()

function n =	getNumBlocks ()
	% data.ABlockedData implementations Implementation of ABlockedData.getNumBlocks More...

function B =	getBlock (nr)
	Implementation of ABlockedData.getBlock. More...

Public Member Functions inherited from data.FileData
	FileData (data_dir)

function	relocate (char new_root)
	Relocates this FileData instance to a different folder. More...

function	delete ()

Public Member Functions inherited from data.ABlockedData
function [ matrix< double > U , matrix < double > S , matrix< double > V ] =	getSVD (integer k,matrix< double > Vexclude,colvec< integer > targetdims)
	Computes an SVD on this blockwise matrix \(A = USV^T\). More...

function prod =	mtimes ()
	Need left-sided matrix multiplication if RHS singular vectors V should be returned. More...

function A =	toMemoryMatrix ()
	Converts this FileMatrix to a full double matrix. More...

Static Public Member Functions
static function data.FileMatrix fm =	recoverFrom (char directory)
	Tries to recover a FileMatrix from a given directory, containing the old block data files. More...

static function bs =	blockSizeOf (arg1,integer arg2)
	Computes the block size in Megabytes for a given matrix of matrix dimensions. More...

static function res =	test_FileMatrix ()

static function res =	test_transposeTimes ()

static function res =	test_SVD ()

static function res =	test_SaveLoad ()

static function res =	test_SumPower ()

static function res =	test_ScalarMult ()

static function res =	test_Times_MTimes ()

static function res =	test_2FileMatrix_MTimes ()
	% Two FileMatrices No blocks More...

static function res =	test_SpeedSVDTransp ()
	Test results for 100000x100 matrix: BlockSVD: Computing truncated 50-SVD on 100x1000000 matrix (3 blocks)... Direct time: 2475.53, Transposed time: 756.023, transposed SVD time: 661.171. More...

static function	test_PartialSVD ()
	Tests the selective SVD/POD block-wise algorithm. More...

Static Public Member Functions inherited from data.ABlockedData
static function res =	test_BlockSVD_vs_SVD ()

Public Attributes
	InPlaceTranspose = false
	Set this flag to true, if only so much space should be required as actually needed when transposing the matrix (i.e. twice the size of the current matrix). More...

integer	bCols
	The number of columns for each block. More...

integer	nBlocks
	The number of blocks into which this matrix is separated. More...

integer	n
	The number of rows. More...

integer	m
	The total number of columns. More...

	MinValue = Inf

	MaxValue = -Inf

Public Attributes inherited from data.FileData
char	Host = ""
	The host machine this file data is created on. More...

char	DataDirectory = ""
	The root folder where the FileData's files are saved. More...

Public Attributes inherited from handle
	addlistener
	Creates a listener for the specified event and assigns a callback function to execute when the event occurs. More...

	notify
	Broadcast a notice that a specific event is occurring on a specified handle object or array of handle objects. More...

	delete
	Handle object destructor method that is called when the object's lifecycle ends. More...

	disp
	Handle object disp method which is called by the display method. See the MATLAB disp function. More...

	display
	Handle object display method called when MATLAB software interprets an expression returning a handle object that is not terminated by a semicolon. See the MATLAB display function. More...

	findobj
	Finds objects matching the specified conditions from the input array of handle objects. More...

	findprop
	Returns a meta.property objects associated with the specified property name. More...

	fields
	Returns a cell array of string containing the names of public properties. More...

	fieldnames
	Returns a cell array of string containing the names of public properties. See the MATLAB fieldnames function. More...

	isvalid
	Returns a logical array in which elements are true if the corresponding elements in the input array are valid handles. This method is Sealed so you cannot override it in a handle subclass. More...

	eq
	Relational functions example. See details for more information. More...

	transpose
	Transposes the elements of the handle object array. More...

	permute
	Rearranges the dimensions of the handle object array. See the MATLAB permute function. More...

	reshape
	hanges the dimensions of the handle object array to the specified dimensions. See the MATLAB reshape function. More...

	sort
	ort the handle objects in any array in ascending or descending order. More...

Public Attributes inherited from data.ABlockedData
double	MinRelSingularValueSize = 1e-20
	The minimum relative value of singular values that triggers selection of the compared to the largest one. More...

Static Public Attributes
static const ::integer	BLOCK_SIZE = 512
	The default block size to use for new FileMatrix instances. [MB]. More...

Protected Member Functions
function this =	saveobj ()

Protected Member Functions inherited from data.FileData
function file =	getfile (file)

function this =	saveobj ()
	Set saved flag so that the data files do not get deleted on the delete method. More...

Static Protected Member Functions
static function this =	loadobj (this, initfrom)
	Loads a FileMatrix instance. More...

Static Protected Member Functions inherited from data.FileData
static function this =	loadobj (this, initfrom)
	Loads a FileData instance. More...

Additional Inherited Members
Protected Attributes inherited from data.FileData
logical	isSaved = false
	This flag indicates that this FileData instance has been stored to disk via the save method somewhere. More...

Constructor & Destructor Documentation

data.FileMatrix.FileMatrix	(	matrix< double >	var,
			varargin
	)

Creates a new file matrix. Possible constructors:

FileMatrix(A): Creates a new file matrix

Parameters

var If a scalar, the row dimension. If a matrix, the file matrix is initialized using the matrix value.

varargin

More optional input arguments, see below.

FileMatrix ( ..., m,

"Dir", Dir_value ] [, "BlockSize", BlockSize_value ] )

Required Parameters for varargin:

m If the var argument was a row dimension, the column dimension \(m\) is a required argument.

Named Parameters for varargin:

Dir A char array denoting the target root folder where the file-containing data folder should be stored. ( Default: KerMor.App.TempDirectory )
BlockSize The maximum block size in MegaBytes. ( Default: KerMor.App.BlockSize )

Definition at line 189 of file FileMatrix.m.

References KerMor.App(), bCols, IDGenerator.generateID(), isposrealscalar(), m, n, nBlocks, numel(), handle.reshape, size(), subsasgn(), and t.

Referenced by data.ApproxTrainData.computeFrom().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Function Documentation

function bs = data.FileMatrix.blockSizeOf	(		arg1,
		integer	arg2
	)

static

Computes the block size in Megabytes for a given matrix of matrix dimensions.

The matrices are assumed to contain double values.

Parameters

arg1	Either a double matrix or the row number
arg2	The column number Default: []

Definition at line 1108 of file FileMatrix.m.

References numel().

Here is the call graph for this function:

function copy = data.FileMatrix.copyWithNewBlockSize ( block_size )

Definition at line 325 of file FileMatrix.m.

References data.FileData.DataDirectory, getBlockPos(), k, m, n, and nBlocks.

Here is the call graph for this function:

function trans = data.FileMatrix.ctranspose ( )

Definition at line 778 of file FileMatrix.m.

References data.FileData.DataDirectory, getBlockPos(), InPlaceTranspose, k, m, n, nBlocks, and t.

Here is the call graph for this function:

function data.FileMatrix.delete ( )

Definition at line 880 of file FileMatrix.m.

References data.FileData.DataDirectory, k, and nBlocks.

function res = data.FileMatrix.eq ( B )

Definition at line 843 of file FileMatrix.m.

References all(), and size().

Referenced by ne().

Here is the call graph for this function:

Here is the caller graph for this function:

function B = data.FileMatrix.getBlock ( nr )

virtual

Implementation of ABlockedData.getBlock.

Implements data.ABlockedData.

Definition at line 908 of file FileMatrix.m.

Referenced by transposedTimes().

Here is the caller graph for this function:

function pos = data.FileMatrix.getBlockPos ( nr )

Returns the column indices of the block "nr" within the full matrix.

Definition at line 314 of file FileMatrix.m.

References bCols, and m.

Referenced by testing.DEIM.computeDEIMErrors(), copyWithNewBlockSize(), ctranspose(), sum(), and transposedTimes().

Here is the caller graph for this function:

function [ rowmin , rowmax ] = data.FileMatrix.getColBoundingBox ( )

Computes the bounding box of the matrix with respect to columns.

Definition at line 298 of file FileMatrix.m.

References n, and nBlocks.

Referenced by approx.algorithms.ABase.computeApproximation().

Here is the caller graph for this function:

function n = data.FileMatrix.getNumBlocks ( )

virtual

% data.ABlockedData implementations Implementation of ABlockedData.getNumBlocks

Implements data.ABlockedData.

Definition at line 897 of file FileMatrix.m.

References n, and nBlocks.

function data.FileMatrix.horzcat ( varargin )

Definition at line 870 of file FileMatrix.m.

function this = data.FileMatrix.loadobj	(	this,
		initfrom
	)

staticprotected

Loads a FileMatrix instance.

Required fields of initfrom:

Generated fields of this:

Definition at line 999 of file FileMatrix.m.

References bCols, InPlaceTranspose, m, MaxValue, MinValue, and nBlocks.

function diff = data.FileMatrix.minus ( B )

Todo:: need FileMatrix as result type here

Definition at line 556 of file FileMatrix.m.

References all(), and size().

Here is the call graph for this function:

function a = data.FileMatrix.mldivide ( R )

Definition at line 591 of file FileMatrix.m.

function AB = data.FileMatrix.mtimes ( B )

Override of ABlockedData.mtimes.

Required fields of B:

Definition at line 610 of file FileMatrix.m.

References k, data.ABlockedData.mtimes(), size(), and sum().

Here is the call graph for this function:

function res = data.FileMatrix.ne ( B )

Definition at line 865 of file FileMatrix.m.

References eq().

Here is the call graph for this function:

function n = data.FileMatrix.numel ( )

Definition at line 434 of file FileMatrix.m.

References n.

Referenced by blockSizeOf(), and FileMatrix().

Here is the caller graph for this function:

function value = data.FileMatrix.power ( expo )

Definition at line 416 of file FileMatrix.m.

References data.FileData.DataDirectory, k, m, n, and nBlocks.

function data.FileMatrix fm = data.FileMatrix.recoverFrom ( char directory )

static

Tries to recover a FileMatrix from a given directory, containing the old block data files.

Parameters

directory The directory to recover from. If not given, KerMor.getDir is used. Default: KerMor.getDir

Return values

fm	The recovered FileMatrix instance. [] if the KerMor.getDir dialog has been aborted.

Generated fields of fm:

Definition at line 1045 of file FileMatrix.m.

References KerMor.getDir(), k, and size().

Here is the call graph for this function:

function data.FileMatrix.relocate ( new_root )

Relocates this FileMatrix instance to a new folder.

Note that the FileMatrix takes a directory inside which it will create a new directory using a certain hash. Thus, upon relocation, the folder that contains the created folder must be passed.

For example, assume we have DataDirectory=/some/path/matrix_246sg351dg. Then relocate with /some/new/root leads to the new path /some/new/root/matrix_246sg351dg at which the blocks of this file matrix are assumed to reside.

See Also: data.FileData.relocate

New in 0.7:: (Daniel Wirtz, 2013-05-28) Added this method.

Parameters

new_root The root folder that contains a FileMatrix self-defined folder used to store the matrix blocks.

Definition at line 335 of file FileMatrix.m.

References data.FileData.DataDirectory.

Referenced by data.ApproxTrainData.relocate().

Here is the caller graph for this function:

function this = data.FileMatrix.saveobj ( )

protected

Definition at line 922 of file FileMatrix.m.

References data.FileData.DataDirectory.

function [ n , m ] = data.FileMatrix.size ( dim )

virtual

Implementation of ABlockedData.size.

Implements data.ABlockedData.

Definition at line 439 of file FileMatrix.m.

References m, and n.

Referenced by eq(), FileMatrix(), minus(), mtimes(), recoverFrom(), subsref(), test_PartialSVD(), test_SVD(), test_Times_MTimes(), and times().

Here is the caller graph for this function:

function B = data.FileMatrix.spawnWithContent ( matrix< double > A )

Creates a new FileMatrix containing the matrix A. The matrix is stored at the at the same location as the current matrix and the same block size is used.

Parameters

A	The new matrix.

Return values

B	A new data.FileMatrix instance containing B.

Definition at line 283 of file FileMatrix.m.

References data.FileData.DataDirectory.

Referenced by data.ApproxTrainData.makeUniqueXi().

Here is the caller graph for this function:

function this = data.FileMatrix.subsasgn	(	key,
		value
	)

Implements subscripted assignment.

See Also: subsasgn

Required fields of key:

Generated fields of this:

Definition at line 509 of file FileMatrix.m.

References m, and nBlocks.

Referenced by FileMatrix().

Here is the caller graph for this function:

function varargout = data.FileMatrix.subsref ( key )

Implements subscripted value retrieval.

See Also: subsref

Required fields of key:

Definition at line 460 of file FileMatrix.m.

References m, n, nBlocks, and size().

Here is the call graph for this function:

function s = data.FileMatrix.sum ( dim )

% Overloaded methods

Definition at line 390 of file FileMatrix.m.

References getBlockPos(), k, m, n, and nBlocks.

Referenced by mtimes(), and test_SumPower().

Here is the call graph for this function:

Here is the caller graph for this function:

function res = data.FileMatrix.test_2FileMatrix_MTimes ( )

static

% Two FileMatrices No blocks

Definition at line 1327 of file FileMatrix.m.

static function res = data.FileMatrix.test_FileMatrix ( )

static

Definition at line 1130 of file FileMatrix.m.

References all(), Utils.getBoundingBox(), k, and t.

Here is the call graph for this function:

function data.FileMatrix.test_PartialSVD ( )

static

Tests the selective SVD/POD block-wise algorithm.

Definition at line 1380 of file FileMatrix.m.

References k, and size().

Here is the call graph for this function:

static function res = data.FileMatrix.test_SaveLoad ( )

static

Definition at line 1206 of file FileMatrix.m.

function res = data.FileMatrix.test_ScalarMult ( )

static

Todo:

need implementation for >1 nBlock matrices and scalar values

need separate test for each overridden operator

Definition at line 1243 of file FileMatrix.m.

function res = data.FileMatrix.test_SpeedSVDTransp ( )

static

Test results for 100000x100 matrix: BlockSVD: Computing truncated 50-SVD on 100x1000000 matrix (3 blocks)... Direct time: 2475.53, Transposed time: 756.023, transposed SVD time: 661.171.

Thus: Auto-transpose for matrices with nBlocks > 1

Definition at line 1351 of file FileMatrix.m.

References t.

static function res = data.FileMatrix.test_SumPower ( )

static

Definition at line 1229 of file FileMatrix.m.

References all(), and sum().

Here is the call graph for this function:

static function res = data.FileMatrix.test_SVD ( )

static

Definition at line 1182 of file FileMatrix.m.

References size(), and t.

Here is the call graph for this function:

function res = data.FileMatrix.test_Times_MTimes ( )

static

Todo:

need implementation for >1 nBlock matrices and scalar values

need separate test for each overridden operator

Definition at line 1275 of file FileMatrix.m.

References all(), Norm.L2(), and size().

Here is the call graph for this function:

static function res = data.FileMatrix.test_transposeTimes ( )

static

Definition at line 1174 of file FileMatrix.m.

References t.

function AB = data.FileMatrix.times ( B )

Override of ABlockedData.mtimes.

Definition at line 739 of file FileMatrix.m.

References k, data.ABlockedData.mtimes(), and size().

Here is the call graph for this function:

function res = data.FileMatrix.transposedTimes ( B )

Implements the operation A'*B for this matrix A and another FileMatrix B.

Required fields of B:

Definition at line 365 of file FileMatrix.m.

References getBlock(), getBlockPos(), m, nBlocks, and t.

Here is the call graph for this function:

function data.FileMatrix.vertcat ( varargin )

Definition at line 875 of file FileMatrix.m.

Member Data Documentation

data.FileMatrix.bCols

The number of columns for each block.

Note: This property has non-standard access specifiers: SetAccess = Private, GetAccess = Public; Matlab documentation of property attributes.

Definition at line 82 of file FileMatrix.m.

Referenced by data.ApproxTrainData.computeFrom(), FileMatrix(), getBlockPos(), and loadobj().

data.FileMatrix.BLOCK_SIZE = 512

static

The default block size to use for new FileMatrix instances. [MB].

This is used as default value in KerMor.App.BlockSize, if not set differently (machine dependent)

The block size unit is Megabyte.

Default: 512MB

Definition at line 49 of file FileMatrix.m.

data.FileMatrix.InPlaceTranspose = false

Set this flag to true, if only so much space should be required as actually needed when transposing the matrix (i.e. twice the size of the current matrix).

If false (default), running ctranspose will create small blocks fitted to the size of the meshed sizes temporarily. This needs 3 times the space instead of 2 but avoids reloading the whole block even if no data is changed where it already exists.

Default: false

Definition at line 66 of file FileMatrix.m.

Referenced by ctranspose(), and loadobj().

data.FileMatrix.m

The total number of columns.