2015#

Release Notes for 2015#

mm-dd#

12-29#

Advance to newer version of cppad and cppad_mixed .
Use new CppAD to_string utility in place off dismod_at/include/to_string.hpp .

12-24#

Advance to newer version of cppad_mixed (rerun bin/install_cppad_mixed.sh ). This fixed a bug in that package.

12-15#

Advance to newer version of cppad (rerun bin/install_cppad.sh ). This should have been done on 12-09 when we converted to using cppad_mixed package.

12-14#

Advance to a newer version of cppad_mixed (rerun bin/install_cppad_mixed.sh ).

12-10#

Advance to a newer version of cppad_mixed (rerun bin/install_cppad_mixed.sh ).

12-09#

First version of dismod_at that has cppad_mixed split out as a separate package. The following definitions are no longer used by the dismod_at cmake command : cppad_mixed_set_sparsity , cppad_mixed_libdir , and cppad_mixed_prefix .

11-23#

The Ipopt optimizer has a bound relaxation factor that caused dismod_at to try and evaluate its object at negative rates. This has been fixed by setting this factor to zero.

11-22#

Move the fit_residual table to the fit_data_subset_table . This is a better name as evidenced by the fact that a separate data_subset_id column is no longer necessary; see dev::fit_data_subset_table::fit_data_subset_id .
Change the previous start_command to have the source argument. The source corresponding to its previous usage is prior_mean . The fit_var option uses the results of one fit as the starting value for the next.

11-21#

Change create_database so that it always uses None (not empty string) when the corresponding database value is null . The effected the smooth_table values for mulstd_value_prior_name , mulstd_dage_prior_name , mulstd_dtime_prior_name . It also effected the rate_table values for parent_smooth , child_smooth .
The bin/import_cascade.py example now outputs its files in the same direction as its option.csv file, (instead of a subdirectory with the same name as the cascade run). This is because the option file is not general purpose and should be in a directory that is special for the particular case.
Created test/ihme/22906.py , a test (under construction) that fits ihme model number 22906.
Change column order in bin/database2csv.py data.csv file so meas_value is next to avgint .
Make the prior_none and prior_zero_one the first two priors so they are easy to reference by their prior_id .
Have test/ihme/22906.py allow for jump between age 30 and 31 (at any time).

11-20#

Improve the new derivative checker.
Improve bin/import_cascade.py .
The fit_command was using the mean for the variable values, instead of the values specified in start_var_table . This has been fixed.
The optimization max_num_iter was changed to allow for zero. This enables one to use the starting values in the fit_var_table and fit_data_subset_table tables.

11-17#

Add another derivative checker cppad_mixed ipopt_fixed_check_grad_f (not yet part of cppad_mixed user API).

11-16#

Add the user_jump_at_age.py example.

11-15#

The fit_var_value column in the fit table has been changed to the fit_var_value column.
Add residual columns for value dage and dtime to the fit_var table.

11-14#

There was a bug when running models with no random effects because the cppad_mixed ran_like was non-empty. This has been fixed. In addition, the cppad_mixed sub-package now detects when this is the case and reports the error. Furthermore, there were some cases where the model for the data did not depend on the random effects, but the Laplace approximation was still being applied (correct but not efficient). This has also been fixed.

11-12#

Add random_seed to the speed tests user_speed.py and cppad_mixed capture_xam.cpp .
Change the log table format for the command message type. In addition, improve documentation of what value is used for the random_seed .
There was a problem with the updating of the version number in bin/setup.py that could cause the python install command to fail. This has been fixed.

11-10#

Group all of the user database documentation under the section database
Modify bin/import_cascade.py to take advantage of the null option for the rate parent_smooth_id and child_smooth_id .
Change from text can be empty to text can be null (to make it more like other types); see Text can be Null, not Empty ).
Implement the space saving Jacobian and Hessian calculations for the cppad_mixed fixed likelihood \(g(\theta)\).
Only initialize the Hessian terms for fixed likelihood \(g(\theta)\) and the cppad_mixed constraint function \(c(\theta)\) when cppad_mixed quasi_fixed is false.
Fix transposition of correspondence between age_id , time_id and var_id the var_table .

11-09#

Advance to a new version of CppAD; i.e., need to rerun bin/install_cppad.sh .
Remove derivative testing from the tests in the speed directory. Increase the number of random effects in cppad_mixed capture_xam.cpp to reduce the change of false negative tests. Increase the number of data points in user_speed.py for the same reason. (This is reasonable because cppad_mixed is now faster and uses less memory.)
The dismod_at python module is now part of the distribution; see Install .
Change the C++ dev::get_table_column.cpp so it will handle upper case for column types. In addition, if a column has an invalid type, report the table and column and invalid type.

11-07#

One can now inform dismod_at that a specific parent rate (child rate effect) is always zero by setting parent_smooth_id ( child_smooth_id ) to null . This reduces the number of model_variables .
The example/user/speed.py test was modified to take advantage of this feature. The memory requirement reported by massif decreased from 21.45 MB to 12.64 MB. The time in Ipopt decreased from 6.142 sec to 2.477 sec the time in NLP function evaluations decreased from 16.760 to 10.321. The number of grid points for each of the child rate effects (random effects) was then increased from one to four.
The random effects sub-package name was changed from mixed_cppad to cppad_mixed .

11-06#

Add lower and upper limits to the random effects in cppad_mixed optimize_random and cppad_mixed optimize_fixed . This is represented by bound_random in the option table.

11-05#

The relrisk and mtwith integrands were not being computed properly. This has been fixed.
The fit results were sometimes missing from the database2csv.py output files (when they should have been there). This has been fixed.

11-04#

Add fit_simulate_index to the option table and change bin/database2csv.py to display simulated data fit_simulate_index is not empty.
Add a user example that uses fit_simulate_index to fit simulated data; see user_fit_sim.py .
The sample_table may be created directly to control the input to the predict_command .
One may now change values in the option table without invalidating all of the output tables .
There was a bug in the solution of the differential equation. The iota_pos_rho_zero and iota_zero_rho_pos rate_case options were reversed. This bug was introduced on 10-29 and has been fixed.
The cppad_mixed ran_like virtual function must be implemented for both a1_double and a2_double (one used to only need to implement a2_double ).

11-03#

Change bin/database2csv.py to only take one argument.
Change all the commands to have syntax

dismod_at file_name command

i.e., switch the order of file_name and command .

11-02#

Added the data base dump program bin/database2csv.py . This is just an example of how one might view the information in the database. It is not documented or tested and may change with time.
Working on improving bin/import_cascade.py , another undocumented example that may change with time.

11-01#

The reduction in memory and increase in speed on 10-30 did not make sense and does not seem to be reproducible (even using the source code check in that was intended to do so).
One can now use log-Gaussian and log-Laplace for the prior on differences for fixed effects; see fixed effects age and time differences. This required a complete re-write (and simplification) of the documentation for the weighted residuals and log-densities. In addition, the actual definitions were changed for the differences; see the difference version of the weighted residuals and log-densities . The corresponding wish list item has been removed.

10-30#

The notation for the weighted residual and log-density was greatly simplified by adding the density as an argument.
The simulated.py speed test was changed to use the log-Gaussian instead of Gaussian density for the measured values.
A large reduction in memory, and an increase in speed, was accomplished by making the transformed standard deviation sigma_i depend the data and not on the model_variables . To be specific, valgrind --tool=massif reported the following reduction for the maximum memory required by simulated.py : old = 21.73 MB , new = 478.8 KB . In addition, Ipopt reported the following total CPU secs in NLP function evaluations: old = 12.98 , new = 8.724 .

10-29#

The change on 10-27 whereby node_rate_value_info was reverted because it used three times as much memory and time. The name was however changed from node_rate_value_info to rate_case to avoid confusion with dev::pack_info_node_rate.cpp . In addition, the correction of the bug in dev::eigen_ode2.cpp was retained.
It is now an error if the value nan appears in the database; see real cannot be nan .
Add tracing, during CppAD’s forward mode, of values in the dev::residual_density.cpp function that should not be negative. This gives feedback as to the cause of CppAD’s message that a forward mode result is nan .
Add a wish_list item.

10-28#

Advance to a newer version of CppAD. The requires that you re-run
```
bin/install_cppad.sh
```
Remove some unused code in the dismod_at user_example examples (used for return value in get_started_db.py ). In addition, simply setting of the standard deviation multipliers to one (in the dismod_at user_example examples).
Add a simulated data example that also serves as a dismod_at speed test (the size of the problem can be easily scaled up).
The dismod_at and cppad_mixed speed tests were placed in the speed directory. They are now run separately from the correctness tests with the command
```
make speed
```
See instructions for running the tests .

10-27#

The values in the prior_table were checked to make sure that lower <= mean <= upper , except for the uniform density case std > 0 , and in the log density cases eta is not null.
There was a bug in the dev::eigen_ode2.cpp routine whereby nan could result even if the rates satisfied their prescribed conditions. This has been fixed.
Modify dev::eigen_ode2.cpp so that it works for all combinations of rates and remove node_rate_value_info from the option_table . Note that this is different from dev::pack_info_node_rate.cpp (which is only in the developer documentation).

10-26#

Changed the specification for the create_database argument option_table . To be specific, the row

{ 'name' : 'parent_node_id' , 'value' :’ parent_node_id ‘ }

was replace by the row

{ 'name' : 'parent_node_name' , 'value' :’ parent_node_name ‘ }

10-25#

Make the choice between quasi-Newton and full Newton a program option for the cppad_mixed package; see cppad_mixed quasi_fixed .
The option to choose between quasi-Newton and full Newton for the fixed effects optimization was moved into the options table; see quasi_fixed .

10-24#

Reduce the probability of the test/cppad_mixed/binomial.cpp test failed by chance during testing .
The cppad_mixed optimize_fixed effects procedure can now uses a BFGS method (and requires much less memory). The cppad_mixed_newton option in the Setup can be used to select the old Newton method.
The cppad_mixed_set_sparsity option was also added to the Setup .
Create a separate documentation section for editing of run_cmake.sh during the install procedure.

10-20#

The approx_mixed sub-package name was changed to cppad_mixed . In addition example names of the form approx_ name where changed to mixed_ name ; e.g., cppad_mixed mixed_derived .
The cppad_mixed_eigen library was not being installed and the bin/check_cppad_mixed.sh test failed. This has been fixed.

10-19#

Added an example that demonstrates that the random effects variance term may cause the solution to move from an exact match with the data values; see cppad_mixed data_mismatch_xam.cpp .

10-16#

Two new tests were added to check cases where the solution for the random effects should be zero; see test/devel/cppad_mixed/zero_random.cpp and test/user/zero_random.py .

10-15#

The instructions in install_unix were modified to make it clear that the external package installers only need to be run once; see Special Requirements .
A separate cppad_mixed ipopt_options was added for the random effect in cppad_mixed optimize_random and cppad_mixed optimize_fixed .
The splitting of the optimization options between fixed and random effects was extend to the dismod_at API; see optimizer options .
Advance to a newer version of CppAD (to avoid another bug) and hence requires bin/install_cppad.sh to be re-run.

10-14#

Use full numerical precision when converting double precision values to strings for writing to output tables (it used to be that only about six digits of precision were preserved).

09-18#

There was an error in the user_no_children.py example (the parent node was world and should have been canada ). This has been fixed. In addition, the fitted values are now checked at the end of the example (so that in the future, this type of mistake will be automatically detected).

09-03#

Advance to a newer version of CppAD (to avoid a bug when used with c++98 compilers) and hence requires bin/install_cppad.sh to be re-run.
The cppad_mixed newton_step function was not taking advantage of the recent memory saving changes to CppAD (using bool_sparsity_enum instead of pack_sparsity_enum ). This has been fixed.
The cppad_mixed newton_step was being recorded, but not used, and hence more memory was used, instead of memory being saved. This has been fixed.
Reduce the chance of a false failure from example/devel/cppad_mixed/user/capture_xam.cpp (takes longer to run).
Print result of test for C++ nullptr during Cmake Command

08-31#

Reduce memory required by cppad_mixed sub-package. This required some recent improvements to CppAD and hence requires bin/install_cppad.sh to be re-run.

08-27#

Reduce the chance of a false failure from example/devel/cppad_mixed/user/capture_xam.cpp (takes longer to run).

08-21#

Edit discussion of text can be empty (now it must be null instead of empty) see Text can be Null, not Empty ).
Remove the special prior that is a constant one from the user_example examples (since 08-12 null can be used to the same effect for standard deviation multipliers ).

08-18#

There was a problem with the eigen install script and its use of the -i flag on Mac systems. This has been fixed.
Suppress certain warnings that occur when compiling with eigen package (need to be fixed by that package). This now requires linking both the cppad_mixed and cppad_mixed_eigen libraries.

08-15#

Improve the cppad_mixed theory section.
Add diagnostics as to the size of the cppad_mixed problem representation; see cppad_mixed size_map .

08-14#

Change the name of the avg_case_table to avgint_table .
Change all covariate value storage to be a vector separate from the corresponding data_table or avgint_table value storage. (see speed comment for 08-13 below).

08-13#

The covariate values storage was changed from a vector for each data row to one vector for all the data rows. For an example with 77,288 rows of data this changed to time to read the entire database from 334 seconds to one second.

08-12#

The var_command was changed back to the init_command (because it creates more than just the var_table ).
There was a bug whereby the standard deviation multipliers were not being used. This has been fixed.
Use null for a standard deviation multiplier now eliminates it from the list of variables; see smooth_table .

08-09#

There was a bug in the var_command when there were covariate multipliers for multiple integrands or multiple rates. This has been fixed and is tested by the user example meas_covariate.py .
The routine dev::data_subset.cpp was changes to pre-allocate its result instead of growing the vector as needed. This resulted in a huge savings (in one test case a 24 minute run time was reduced to 3.5 minutes).
The dev::get_db_input.cpp routine now checks the lower and upper limits implied by the choice of node_rate_value_info . This was removed on 2015-10-27 .

08-08#

Change the init_command to the var_command (because it creates the var_table ).
Change the measurement covariate multiple type rate_mean to rate_value .
Simplify the node_rate_value_info choice in the option table.
Fix logging an error message that has the single quote character ' in it.
Improve the usage message that is printed when dismod_at is executed with no command arguments.
Improve the error message when the mulcov_table has an invalid value in the mulcov_type column.
Reduce the chance of a false failure from test/binomial_test.cpp .

08-04#

Some compiler errors (on older compilers) were added. In addition, the minimal version of sqlite3 was specified.

08-03#

First version that actually installs the cppad_mixed sub-package using the cppad_mixed_libdir option to the Cmake Command .
The cmake command script was not using bash as its shell for all systems. This has been fixed.
Reduce the chance of random failure in test/devel/n_mixture.cpp (using more simulated data).

08-02#

Remove the option to include Laplace likelihoods (absolute value terms) in the cppad_mixed random negative log likelihood . This reduces the number of necessary recordings and corresponding memory.
The cppad_mixed ran_like function was changed from use a3_double objects to a2_double objects.
Add the cppad_mixed capture_xam.cpp example and test. This is both a correctness and speed test.

08-01#

The Hessian w.r.t. \(\beta\),

\[H^{(2)}_\beta H( \beta, \theta, u)\]

is no longer recorded as a separate AD function, but is rather evaluated using the recording for \(H( \beta , \theta , u)\) (instead of its own separate recording. The greatly reduced the program memory requirements.
The cppad_mixed ran_like function was changed from use a5_double objects to a3_double objects.

07-29#

The term ‘prior negative log-likelihood’ was change to cppad_mixed fixed likelihood Fixed Likelihood, g(theta) / .
The term ‘joint negative log-likelihood’ was change to cppad_mixed random likelihood Random Likelihood, f(theta, u) / .

07-28#

Some example were missing, or were including the wrong source code. This has been fixed in the following user examples: cppad_mixed no_random_xam.cpp , create_database.py . It was also fixed for the following developer examples: dev::avgint_subset_xam.cpp.cpp , dev::subset_data_xam.cpp.cpp .
Handle the case where the cppad_mixed fix_like is empty.

07-27#

Grouped all the user level cppad_mixed examples in the section cppad_mixed user .

07-25#

Add the dev::cpp_create_table.cpp routine, and replace all uses of dev::put_table_row.cpp by create_table (much faster).

07-24#

Add the derivative_test option (because it is to slow to do all the time).
Fix the automatic setting of the number of age and time points in the ode_grid.

07-23#

Add the user_lasso_covariate.py example / test.
Fix some bugs related to using laplace and log_laplace densities .
Detect and report an error in the data table when meas_std is less than or equal zero and when density_id corresponds to a uniform distribution.

07-22#

Add the user_no_children.py example.
Exit with an error if cppad_mixed optimize_random is called in a case where there are no random effects.
There was a bug in the case where the second derivative of the random part of the objective \(H( \beta , \theta, u)\) with respect to \(\beta\) was zero. This has been fixed.
The data corresponding to the Parent Node is now in represented using cppad_mixed p(z | theta) instead of cppad_mixed p(y | theta, u) . This speeds up the computation and enables one to use Laplace and Log-Laplace likelihoods for this data (with out affecting the Laplace approximation for the integral).

07-21#

The cppad_mixed sub-package was extended to work in the case were there are cppad_mixed no random effects .

07-20#

If random_seed is zero, use the unix_time in the log table for the random seed.
Remove the value message_type from the log table (not yet necessary due to change above).
Simplify the command start and done messages.
Include data that is independent of the random effects cppad_mixed z in the discussion of the cppad_mixed cppad_mixed problem . Also fix a \(\B{p} ( y , \theta )\) that was written as \(\B{p} ( y | \theta )\) in the problem discussion.

07-19#

The name of the argument_table was change to the option_table . Some bugs were fixed in the cppad_mixed constraint option.
The print_level option was added to control tracing during optimization, use no tracing (zero) as default.
There was a bug in the values of age_id and time_id in the var table. This has been fixed.
Change the definition of dage_prior_id and dtime_prior_id so that they refer to differences and not difference quotients because differences are simpler.
The user_diff_constraint.py example was added to documentation and test suite.
The solution check failed warning would sometimes print incorrectly when a fixed effect was at its lower or upper bound. This has been fixed.

07-17#

First version of the cppad_mixed sub-package that works with cppad_mixed constraints ; e.g., see cppad_mixed mixed_constraint_xam.cpp .
The solution check failed warning was printing when the check actually passed (since 07-15). This has been fixed.
Add the value message type to the log table (it has since been removed).

07-16#

In the cppad_mixed sub-package, changed joint_density to cppad_mixed ran_like and changed prior_density to cppad_mixed fix_like .

07-15#

Change create_database arguments to end in _ dict when they are a list of dictionaries, so similar to table_dict .
When writing to standard error, add Error: (Warning: ) to beginning of error messages (warning messages).
The init_command no longer deletes the current fit, fit_data_subset, truth_var, simulate, sample and predict tables. These table get overwritten when the corresponding command is executed; see output tables .

07-14#

Add hold_out to the data table and exclude the flagged data during the fit_command .
Add the fit_data_subset_table and compute it during a fit_command .

07-13#

Record the start of and end of each command in the log_table .
Once there is a database dev::open_connection , use log_table to record all error messages before exiting.
The pure virtual function cppad_mixed fatal_error was added to the cppad_mixed derived class. This enabled dismod_at to log the cppad_mixed error messages.

07-11#

Add an error message when the text value in a table is empty; see Text can be Null, not Empty . Allow for null integer values; see Integer .
The example dev::sim_random_xam.cpp.cpp was missing from documentation. Increase sample_size in sim_random_xam.cpp (to reduce change of random failure). In addition, print random seed when test does fail.
Use null instead of -1 in the following cases: There is no parent node, there is no dage_prior_id , there is no dtime_prior_id .
Add the constraint that a Foreign Key must appear in the corresponding primary key column, or it must be null .

07-10#

First version of predict_command that passes its example / test predict_command.py .
List addition input and output tables in data_flow section.

07-09#

The sample table column sample_value was changed to var_value .
The avgint_table was added to the set of input tables.
The avgint_subset_table was added as an output of the init_command .

07-08#

Change the var_command to init_command and use this command to initialize output tables whenever the input tables change.
Add the data_subset_table .

07-07#

First version of sample_command that passes its example / test.

07-05#

First version of the simulate_command that passes its example / test.
Put the command arguments in the argument_table instead of on the command line (except for the file name of the database).
During install_unix , there was a problem running make check without first running make . This has been fixed.
Use a var_command to create the var_table and put the fit_command results in a separate fit_var_table .

07-04#

Remove the python pack_info class (no longer necessary now that var_table has more information).
Move the get_started.py example to fit_command.py .
Modify the fit_command.py example so that it demonstrates how to use the new var_table information and so that it can be run as is (without configuration).

07-03#

Remove the variable_name from the var_table , add extra columns to specify the exact variable using foreign keys .
Change the mean_mean covariate multiplier type to meas_value .
For cov_type equal rate_value , meas_value , and meas_std , change

cov_type _ mulcov_n_cov -> mulcov_ cov_type _ n_cov

cov_type _ mulcov_info -> mulcov_ cov_type _ info

see mulcov_rate_value , mulcov_meas_value and mulcov_meas_noise .

07-01#

Simulating random numbers (for the simulate_command which is not yet working) has added the gnu scientific library gsl the system requirements.
The Foreign Key convention was added to the documentation.

06-30#

The python routine create_table now adds the unique constraint to name columns . This in turn is used by, and does the same for, create_database . In addition, the C++ examples for dev::devel_table were changed to include constraints; e.g., dev::get_integrand_table_xam.cpp.cpp .
There was a bug in reading the data tables whereby the following error would be incorrectly generated:

table_name _ id must start at zero and increment by one.

Detected in table_name table at table_name _ id = index .

where table_name is any table name. This has been fixed.
Added the simulate_command (not yet working).
The values n_age_ode and n_time_ode are now computed automatically have been removed from the fit_command argument list.
It is no longer necessary to run a separate make command to build the dismod_at executable and library when installing dismod_at. In addition, running the tests is now optional.

06-29#

Add the node_rate_value_info argument to the fit command. (This was removed on 2015-10-27 .) This enables taking advantage of cases where rho or iota are zero (to speed integration). In addition, it allows for some cases where chi is zero.
Add a simple introduction to the Dismod_at Ordinary Differential Equation
Create links from the index (and search) to the rates . In addition, make the definition of the rates more precise; e.g., see iota .
The integrand_name incidence has been changes to Sincidence to emphasize that it is relative to the susceptible population. In addition the new integrand Tincidence has been added (which is relative to the total population).

06-27#

All of the prior_id columns in the smooth_table have been changed to end in _prior_id . To be explicit, the following column names have been changed:

Old name

New name

mulcov_value

mulcov_value_prior_id

mulcov_dage

mulcov_dage_prior_id

mulcov_dtime

mulcov_dtime_prior_id
Update the wish_list

06-26#

Remove the requirement that the age and time values must be in increasing order in the corresponding tables.
The install instructions have been improved.
There was a bug in the dev::weight_info.cpp constructor when the ages or times were not increasing. This was fixed and a second version for this date was created.

06-25#

Change variable_name for measurement covariate and rate covariate multipliers from empty string to a descriptive name. (This column of the var table was later removed.)
The C++11 nullptr feature was not being suppressed when using a compiler that does not support this feature. This has been fixed.
There were some non-standard references, in dev::open_connection.cpp , cppad_mixed ipopt_xam , and dev::get_column_max_xam.cpp.cpp , that gave compiler errors on some systems. This has been fixed.
The particular settings for installing on the IHME cluster have been included as comments under the install dismod_at Setup heading.

06-24#

There was a bug in the create_database routine, to be specific, it was not calling commit and close at the end; see python sqlite.
Also close the connection at the end of each of the python sqlite examples; e.g., at the end of age_table.py .
Change variable_name for the smoothing standard deviation multipliers from empty string to a descriptive name. (This column of the var table was later removed.)

06-23#

The fit_command was changed so that there can only be on set of results in the data base. This changed the contents of the fit_arg_table and the var_table .

06-22#

First version of dismod_at is available to be downloaded and installed; see install_unix .

Old name	New name
`mulcov_value`	`mulcov_value_prior_id`
`mulcov_dage`	`mulcov_dage_prior_id`
`mulcov_dtime`	`mulcov_dtime_prior_id`