2 % Helper
class used to store and restore @ref
DataTree "data tree" objects at
3 % specified checkpoints.
5 % This
class can be used for checkpointing. The most prominent use case is
6 % the ReducedModel.gen_detailed_data() method. When
this method is called,
7 % checkpoints are created after each bigger basis extension step at which the
8 % generated detailed_data is stored. Each of these Afterwards the checkpoint can be re-read
9 % from the harddrive like
this:
11 % ids = Checkpoint.get_ids(rmodel)
12 % [detailed_data, cp] = Checkpoint.restore(rmodel, ids(1),
'latest');
15 % Instead of
'latest' one could also have used a file index. See
16 % #file_rotation_size and store() for more details on this.
19 properties (Hidden=true, Constant)
20 % list of values ignored for hash computation by compute_hash()
21 ignorelist = {
'Mmax',
'MM',
'Mstrich',
'ei_Mmax',
'Mmax',
'M',
'N',
'Nmax',...
22 'verbose',
'debug', ...
23 'enable_error_estimator'...
28 % persistent value
for all
Checkpoint instances controlling whether
29 % checkpoint is enabled or not.
30 enable_auto_restore = 1;
32 enable_storing =
true;
35 properties (Access =
public)
37 % maximum number of files stored in one directory in parallel.
39 % If
this limit is reached, the oldest file is deleted before a
new
41 file_rotation_size = 5;
43 % The level indicates which level in a data tree of
DataTree.
INode instances
44 % can be (re)stored by this
Checkpoint instance.
46 % Level '1' corresponds to the root element of the data tree, level '2' to
47 % a child of the root, level '3' to a child of a child of the root element,
51 % A cell array of indices as returned by
DataTree.INode.get_index() defining
52 % the index of the stored
DataTree.INode instance with respect to the root
53 % element of the tree.
55 % @todo: Why do I not store a single vector
58 % a cell array of arbitrary userdata stored with each
DataTree.
INode node.
61 % a cell array of
id strings which describes the
DataTree.
INode instance or the
62 % algorithm which generates it.
65 % a cell array of filenames where the
DataTree.
INode children are stored.
71 function cp = Checkpoint(old, index)
72 %
function cp = Checkpoint(old, index)
73 % creates a
new checkpoint
76 % old: an
object of type .Greedy.Checkpoint whose checkpoint data is copied or
77 % included in the newly created checkpoint.
78 % index:
if this is empty, argument old is copied, otherwise a
new
79 % Checkpoint of level old.level+1 is created with data fields.
85 cp.level = old.level + 1;
87 if cp.level <= length(old.userdata)
88 cp.userdata = old.userdata;
90 cp.data_index = old.data_index;
91 cp.filenames = old.filenames;
93 cp.userdata = [ old.userdata, {[]} ];
94 cp.ids = [ old.ids, {''} ];
95 cp.filenames = [ old.filenames, {''} ];
96 cp.data_index = [ old.data_index, {[old.data_index{end}, index]} ];
101 function ec =
get.enable_auto_restore(
this)
102 ec = this.static_flags(
'auto_restore');
105 function this = set.enable_auto_restore(
this, nec)
106 ec = this.static_flags(
'auto_restore', nec);
109 function ec =
get.enable_storing(
this)
110 ec = this.static_flags(
'storing');
113 function this = set.enable_storing(
this, nec)
114 ec = this.static_flags(
'storing', nec);
117 function dd = restore_detailed_data(cp, model, cp_index)
123 tmp = load(fullfile(fp, cpfn));
124 if length(cp.ids) == 1
125 dd = tmp.detailed_data;
128 tmp = load(fullfile(fp, cp.filenames{1}));
129 dd = tmp.detailed_data;
130 for i = 2:length(cp.ids)
131 if ~isempty(cp.ids{i})
133 tmp = load(fullfile(fp, cp.filenames{i}));
134 set(dd, cp.data_index{i}, tmp.detailed_data);
140 function ret =
get(
this, field,
default)
141 %
function ret =
get(
this, field,
default)
142 % access
function for user data stored in a
Checkpoint instance
145 % field: name of a field in the #userdata structure of
this #level.
146 %
default: This value is returned
if the field does not exist in the
147 % #userdata structure.
150 % ret: This returns the value of the userdata field.
152 ud = this.userdata{this.level};
153 if isfield(ud, field)
160 function child = child(this, index)
161 % function child = child(this, index)
163 % instance shall be generated.
165 % The child must be a
DataTree.INode itself.
168 % index: a scalar indicating the father child-relationship. child must
169 % be the 'index'-th child of father.
172 % child: the created instance of type
Checkpoint
176 % function ret = subsref(this, S)
177 % if isequal(S(1).type, '.') && isequal(S(1).subs, 'ud') ...
178 % && length(S) == 2 && isequal(S(2).type, '.')
179 % if isfield(this.userdata, S(2).subs)
180 % ret = this.userdata.(S(2).subs);
185 % ret = builtin('subsref', this, S);
189 function cp = store(cp, model, detailed_data,
id, ud)
190 % function cp = store(cp, model, detailed_data,
id, ud)
191 % stores a
DataTree.INode tree on the harddrive and attaches a descriptive
192 % text and user data.
194 % The directory where the data is stored is computed via
197 % @note The stored
DataTree.INode knows its fathers, such that it can be
198 % embedded in a full
DataTree.INode tree by restore().
202 % detailed_data: an
object of type
DataTree.INode to be stored on the harddrive
203 %
id:
id strings which describes the
DataTree.INode instance or the
204 % algorithm which generates it.
205 % ud: arbitrary user data to stored with this checkpoint.
219 dates = zeros(length(files),1);
220 if length(files) >= cp.file_rotation_size + 2
221 for i = 1:length(files)
222 if isempty(setdiff(files(i).name, {
'.',
'..'}))
225 dates(i) = files(i).datenum;
228 [dates, ind] = min(dates);
229 cpfn = files(ind).name;
231 cpfn = ['cp', num2str(length(files)-2)];
234 cp.ids{cp.
level} = id;
235 cp.userdata{cp.level} =
structcpy(cp.userdata{cp.level}, ud);
236 cp.filenames{cp.level} = cpfn;
237 save(fullfile(fp, cpfn),
'detailed_data',
'cp');
245 function [dd, cp] = restore_latest_if_available(model,
id)
246 fp =
Greedy.Checkpoint.filepath(model,
id);
247 if isempty(dir(fp)) || ~
Greedy.Checkpoint.static_flags('auto_restore')
251 [dd, cp] =
Greedy.Checkpoint.restore(model,
id, 'latest');
255 function [dd, cp] = restore(model,
id, cp_index)
256 % function [dd, cp] = restore(model,
id[, cp_index])
257 % restores a check point by a descriptive text and a checkpoint index
260 %
id: the descriptive
id string used when storing the
DataTree.INode with
262 % cp_index: an file index between '1' and
#file_rotation_size or the
263 %
string 'latest'. (Default =
'lastest')
266 % dd: restored
object of type
DataTree.
INode . Note, that is the root of a
267 % data tree even
if the
'id' argument refers to a child of it.
279 error(['Could not find checkpoint! Directory ', fp, ' does not exist.']);
282 cpfn =
Greedy.Checkpoint.filename(model,
id, cp_index);
284 tmp = load(fullfile(fp, cpfn));
287 dd = restore_detailed_data(cp, model, cp_index);
291 function cpfn = filename(model,
id, cp_index)
292 fp =
Greedy.Checkpoint.filepath(model,
id);
296 if isequal(cp_index, 'latest')
297 dates = zeros(length(files),1);
298 for i = 1:length(files)
299 if isempty(setdiff(files(i).name, {
'.',
'..'}))
302 dates(i) = datenum(files(i).date);
305 [dates, ind] = max(dates);
306 cpfn = files(ind).name;
308 cpfn = ['cp', num2str(cp_index)];
312 function clear_old_checkpoints(model, num_left)
313 %
function clear_old_checkpoints(model [, num_left])
314 % clears data files in a checkpoint directory
316 % The filepath is generated from the argument
'model'.
320 % num_left: number of files to be left in the directory. (Default = 0)
328 base = fullfile(fp, descr.name);
329 if exist(base,
'dir')
332 dates = zeros(length(dirs),1);
333 for i = 1:length(dirs)
334 if isempty(setdiff(dirs(i).name, {
'.',
'..'}))
337 dates(i) = datenum(dirs(i).date);
340 [dates, inds] = sort(dates, 1, 'descend');
342 for i = inds(num_left+1:end)
343 rmdir(fullfile(base, dirs(i).name), 's')
348 function fp = basepath()
349 % function fp = basepath()
350 % returns the basepath for storage of
Greedy.Checkpoint instances on the hdd.
353 % fp:
string of file path to base directory.
354 fp = fullfile(rbmatlabtemp, 'checkpoints');
357 function fp = filepath(rmodel,
id)
358 % function fp = filepath(rmodel,
id)
359 % computes a file path from the rmodel hash and an
id
363 %
id:
id strings which describes the
DataTree.INode instance or the
364 % algorithm which generates it.
367 % fp:
string of file path to the directory where data is stored.
369 descr = rmodel.descr;
370 mhash =
Greedy.Checkpoint.compute_hash(descr);
371 fp = fullfile(
Greedy.Checkpoint.basepath, descr.name, mhash,
id);
374 function ids = get_ids(rmodel)
375 % function ids = get_ids(rmodel)
376 % returns a list of possible 'ids' to restore for a reduced model.
382 % ids: a cell array of possible ids to restore.
389 ids = setdiff(arrayfun(@(x) x.name, dirlist, 'UniformOutput', false), {
'.',
'..'});
393 function mhash = compute_hash(descr)
394 %
function mhash = compute_hash(descr)
395 % computes an md5 hash
string from the model description used to identify
396 % it over a file name.
399 % descr: a structure describing the model.
402 % mhash: a
string of the hash key computed.
403 il = [Greedy.Checkpoint.ignorelist, descr.mu_names];
404 if isfield(descr,
'filecache_ignore_fields_in_model')
405 il = [ il, descr.filecache_ignore_fields_in_model ];
407 fns = setdiff(fieldnames(descr), il);
410 for i = 1:length(fns)
413 mhash = hash(mstr,
'md5');
416 function str = stringify(x)
417 %
function str = stringify(x)
418 % helper
function to make strings out of function_handle, cell and other objects.
420 % This
function is utilized by the compute_hash() method.
423 % x: an arbitrary
object from which a
string shall be computed.
431 case 'function_handle'
434 str = mat2str(cell2mat(cellfun(@
Greedy.Checkpoint.stringify, x, 'UniformOutput', false) ) );
436 str = 'structs_not_handled_yet';
437 otherwise %
double, logical, uint??,
int??, ...
445 methods (Static, Access = public)
447 function ec = static_flags(flag, nec)
450 pec = struct('auto_restore', true, 'storing', true);
static function fp = basepath()
returns the basepath for storage of Greedy.Checkpoint instances on the hdd.
static function str = stringify(x)
helper function to make strings out of function_handle, cell and other objects.
Helper class used to store and restore data tree objects at specified checkpoints.
function Greedy.Checkpoint cp = store(model,DataTree.INode detailed_data, id, ud)
stores a DataTree.INode tree on the harddrive and attaches a descriptive text and user data...
Interface for a node in a DataTree.
level
The level indicates which level in a data tree of DataTree.INode instances can be (re)stored by this ...
static function fp = filepath(IReducedModel rmodel, id)
computes a file path from the rmodel hash and an id
This is the interface for a reduced model providing methods to compute low dimensional reduced simula...
function s1 = structcpy(s1, s2)
copies the fields of structure s2 into structure s1. If the field to be copied does not exist in s1 y...
Customizable implementation of an abstract greedy algorithm.