--------------------------------------------- lines 6-797 of file: xrst/whats_new/2021.xrst --------------------------------------------- {xrst_begin 2021} {xrst_spell cen cmake covid cygwin dd dir distutils eigen ihme makedirs mkpath mm msg msys pdf podman stdcv stderr stdout sys table table util } Release Notes for 2021 ###################### mm-dd ***** 12-31 ===== An error message in :ref:`db2csv_command-name` was missing its assert. This has been fixed. In addition, all the ``sys.exit(msg)`` calls were changed to ``assert False, msg`` to make then easier to debug. 12-30 ===== Improve the error message that prints the rates when an integrand can not be computed; e.g., prevalence can not be computed when S and C are zero. 12-27 ===== The system_command_prc routine used to print the error message when :ref:`system_command_prc@return_stderr` was false and :ref:`system_command_prc@file_stderr` was None. Now it puts the error message in the assertion that gets generated. 12-22 ===== Fix x-axis tick labels in :ref:`plot_data_fit-name` plots. 12-14 ===== #. Add the :ref:`user_plot_curve.py-name` example and fix the title in :ref:`user_plot_rate_fit.py-name` , :ref:`user_plot_data_fit.py-name` . #. **API Change** : Move :ref:`plot_rate_fit@rate_set` to the last ``plot_rate_fit`` argument. #. The :ref:`system_command_prc-name` function now puts quotes around command line arguments that has spaces (when it prints a command). #. The :ref:`plot_rate_fit-name` and :ref:`plot_data_fit-name` routines now have a command line interface in ``dismodat.py`` . 12-09 ===== Add the *file_stdout* , *file_stderr* , and :ref:`system_command_prc@write_command` arguments to ``system_command_prc`` . 12-08 ===== The ``distutils`` module has been deprecated. Replace all use of ``distutils.dir_util.mkpath`` with ``os.makedirs`` . 12-05 ===== #. Add the :ref:`perturb_command-name` . This is useful for the cascade where the mean of the prior is sometimes very close to the optimization estimate. #. The :ref:`csv2db` The :ref:`modify` were deprecated. 11-29 ===== In plot_curve, add the assumption that the plot data :ref:`values` are non-negative and not all zero. 11-27 ===== #. Add the :ref:`dock_dismod_at.sh@driver` setting to dock_dismod_at.sh so it can use either docker or podman. #. Advance the dock_dismod_at.sh :ref:`dock_dismod_at.sh@dismod_at_version` to 20211121. This requires re-running ./ ``dock_dismod_at.sh image mixed`` because the version of cppad_mixed changed. 11-19 ===== Add the :ref:`plot_curve-name` utility. 11-16 ===== Advance to cppad_mixed-20211025. This improves the :ref:`fit_command@warm_start` option in the fit command. 11-15 ===== Change :ref:`plot_data_fit-name` so that it plots with respect to age and time (as well as index). 11-14 ===== #. The :ref:`connection_file-name` function was added. #. The :ref:`user_average_integrand.py-name` and :ref:`user_csv2db.py-name` examples were added to the :ref:`user_example-name` list. #. Improve the error messages in :ref:`get_name_type-name` . 11-07 ===== #. Make the :ref:`plot_data_fit-name` arguments *integrand_list* and *plot_title* optional. In addition, its return value *n_fit_dict* was changed from a list to a dictionary. #. **API Change** : The changes on 11-07 and 11-06 are API changes to ``plot_data_fit`` . 11-06 ===== #. The :ref:`plot_data_fit-name` routine would crash when one of the integrands had no data. This has been fixed. #. The *plot_title* argument was added to the :ref:`plot_rate_fit-name` and :ref:`plot_data_fit-name` functions. In addition, the *max_plot* argument was added to ``plot_data_fit`` . 11-05 ===== #. The ``plot_data_fit`` function now plots data that is held out in :ref:`plot_data_fit@Plot Legend@Green` . #. **API Change** : Split the *out* column in the data.csv into the :ref:`db2csv_command@data.csv@d_out` and :ref:`db2csv_command@data.csv@s_out` columns. 11-03 ===== Fix bug in :ref:`plot_rate_fit-name` . To be specific, if more that one page of times (of ages) was needed to plot a rate as a function of age (time), only the last page was included in the pdf output file. 10-31 ===== There was a bug in the calculation of :ref:`bnd_mulcov_table@max_cov_diff` during an :ref:`init_command-name` . This has been fixed. 10-25 ===== #. Use the term standard error (not the rate name) to label standard errors plotted by :ref:`plot_rate_fit-name` . #. Print the plot location at the end of :ref:`user_plot_rate_fit.py-name` and :ref:`user_plot_data_fit.py-name` . 10-24 ===== The following arguments were added to :ref:`system_command_prc-name` : *print_command* , *return_stdout* , *return_stderr* . The default settings for these arguments makes ``system_command_prc`` backward compatible. 10-23 ===== Balancing :ref:`hold_out_command@Balancing@Child Nodes` was added to the hold_out command. 10-15 ===== The :ref:`bnd_mulcov_command-name` was not affecting the limits of the covariate multipliers. This has been fixed. 10-14 ===== Add more error detection and reporting to :ref:`get_name_type-name` and :ref:`plot_rate_fit-name` . 10-12 ===== An error report for null covariate :ref:`covariate_table@reference` values was added. In addition, using nulls in undocumented cases was explicitly forbidden; see :ref:`database@Null` . 10-11 ===== Add the :ref:`plot_rate_fit-name` python function. 10-07 ===== Add the :ref:`plot_data_fit-name` python function. 10-06 ===== Add a documentation to the :ref:`user_no_children.py-name` example. 09-01 ===== If the file name for a database in a :ref:`command-name` did not exist, most commands gave the error message: | |tab| ``Could not find table or column in table.`` | |tab| ``detected in age table`` This has been changed to ``Cannot find the database`` *file_name* where *file_name* is the name used for the database. 08-30 ===== The hold_out command did not properly detect and report when the :ref:`hold_out_command@integrand_name` was not valid. This has been fixed. 08-25 ===== Advance dock_dismod_at.sh from version 20210812 to version 20210825. 08-24 ===== Change the some python exits with an error message to raising an assert exception with the same message. (This is more useful in the python debugger.) 08-23 ===== The was a bug in checking that the child nodes in the :ref:`nslist_pair_table-name` were children of the parent node, when the :ref:`node_table@node_id` for the parent node was not zero and the parent node was specified using :ref:`option_table@Parent Node@parent_node_name` . This has been fixed. 08-22 ===== Change :ref:`hes_fixed_table-name` so that it does not include fixed effects that have equal lower and upper limits. 08-21 ===== The ``sim_data`` utility was moved to :ref:`average_integrand-name` . It is still considered under construction. 08-18 ===== #. Change the ``system_command_prc`` function so its :ref:`system_command_prc@result` is a python string (instead of bytes); e.g., see :ref:`user_trace_init.py-name` . #. If an error occurs during :ref:`system_command_prc-name` , print stderr as a string (instead of bytes) and raise an ``AssertionError`` instead of exiting the current program. #. Add documentation in avgint table discussing when :ref:`avgint_table@node_id` should be null. 08-17 ===== There was a problem using a mulcov integrand in the :ref:`avgint_table-name` (the integrand name begins with ``mulcov_`` ). To be specific, these integrands would not get included in the predict table :ref:`predict_table@Avgint Subset` . This has been fixed. 08-15 ===== Add the :ref:`replace_table-name` utility. 08-12 ===== #. Advance to cppad_mixed-20210812. This fixes an install problem in cppad_mixed-20210714. #. Fix the :ref:`system_command_prc@Syntax` in the python ``system_command_prc`` function. To be specific, stderr is not returned. #. Advance dock_dismod_at.sh from version 20210624 to version 20210812; see the items between 06-24 and 08-12 in this section. 08-08 ===== {xrst_comment Merge std branch} #. **API Change** : The transformed standard deviation sigma_i is now computed before the standard deviation adjustment, hence :math:`\sigma` no longer depends on the :ref:`model_variables-name` . As a consequence for :ref:`density_table@Notation@Log Scaled` densities, the :ref:`adjusted standard deviation` is now in log of the data space (it used be have the same units as the data). This fixes a problem with simulating data that has :ref:`mulcov_table@mulcov_type@meas_noise` covariate multipliers. #. Change the definition of meas_delta in data.csv so that it is still in data units even when the adjusted standard deviation is in log of data space. #. Correct the title in the :ref:`user_hes_random.py-name` example. #. Test the :ref:`user_fit_meas_noise.py-name` example using more values for the :ref:`option_table@meas_noise_effect` . #. Move ``user_sample_simple.py`` to :ref:`user_sample_asy_sim.py-name` . 07-29 ===== The Mac changed how ``bash`` worked and this broke it's install. To be specific, ``export DYLD_LIBRARY_PATH`` no longer works (export works for most shell variables but not others). The ``bin/get_cppad_mixed.sh`` was changed to adapt to this problem including advancing to cppad_mixed-20210714. In addition, the :ref:`install_settings.py@specific_compiler` setting was added to ``bin/run_cmake.sh`` to make it easier to build using MacPorts. 07-25 ===== Fix a bug in the :ref:`system_command_prc-name` error reporting. To be specific, when an error occurred during a system command, the command return code was not properly reported. 07-24 ===== #. The ``sim_data`` utility is under construction. It which is slower and easier to use than the :ref:`simulate_command-name` . It also can be used for independent testing of the dismod_at integrators. #. Remove cen_log_students from the :ref:`log scaled density list` (it is not yet a valid density). 07-22 ===== #. Group the :ref:`fit_command@Output Tables` in the fit command documentation. #. Add the :ref:`hes_random table` table to the output of the fit command. #. The default value for :ref:`option_table@trace_init_fit_model` was changed to ``false`` and it was added to the :ref:`option_default-name` list. 07-14 ===== #. Advance to cppad_mixed-20210714. This enabled adding the :ref:`option_table@trace_init_fit_model` option; see :ref:`user_trace_init.py-name` for an example. #. Change ``bin/get_cppad_mixed.sh`` so that :ref:`example_install.sh-name` so that it automatically replaces all the :ref:`install_unix@Special Requirements` . This catches changes in the install of eigen, ipopt, and cppad without have to remove the old install directory. #. Change :ref:`system_command_prc-name` so that it return standard output and prints standard error when an error occurs. 07-05 ===== Fix two bugs in the changes on 07-04. One occurred when using :ref:`option_table@Zero Sum Constraints` . The other was in the :ref:`db2csv_command-name` (the python code value ``true`` was corrected to ``True`` ). 07-04 ===== If a child node does not have any data, its rate random effects automatically use a random bound of zero; see :ref:`option_table@Optimize Random Only@bound_random` . This removes them from the optimization problem and yields the same results. The variable.csv column ``m_bnd`` was changed to :ref:`db2csv_command@variable.csv@bound` to included this new feature in its output. 07-02 ===== There was a bug in the :ref:`set option` command. To be specific, if you set an option that was not yet in the table to the empty string, an empty string (instead of null) was put in the option table. This has been fixed. 06-24 ===== #. Add the :ref:`data_density_command-name` and include its settings in the data.csv file columns :ref:`db2csv_command@data.csv@density` , :ref:`db2csv_command@data.csv@eta` , :ref:`db2csv_command@data.csv@nu` . #. Remove the log density wish list item which is no longer necessary after the improvement to log densities on :ref:`2021@mm-dd@05-20` . #. Improve the usage message with ``dismod_at`` is executed with no arguments. #. Advance to version 20210624 of cppad_mixed and dock_dismod_at.sh. 06-22 ===== #. Add the :ref:`bnd_mulcov_table-name` , :ref:`bnd_mulcov_command-name` and columns :ref:`db2csv_command@variable.csv@m_diff` , and ``m_bnd`` to variable.csv #. Change the section name ``user`` to :ref:`user_example-name` and sort the user examples. 06-18 ===== #. Change the :ref:`compression intervals` from age and time vectors to age and time scalars (use the same intervals for all integrands). #. Sort the table in the :ref:`data_flow-name` section. 06-17 ===== Add options that :ref:`compress intervals` . Include this setting in the :ref:`db2csv_command@data.csv@age_lo` , age_up, time_lo, time_up columns in the data.csv file output by :ref:`db2csv_command-name` . For an example; see :ref:`user_compress.py-name` . 06-15 ===== #. Move the default option values from :ref:`option_table-name` to :ref:`option_default-name` section. #. Re-order and improve the headings in the :ref:`option_table-name` documentation. #. Include :ref:`hold_out_command-name` and :ref:`init_command-name` in the data flow :ref:`command output table` . In addition, add some missing commas in that table. #. Improved documentation for :ref:`fit_command@hold_out` . #. Fix bug in :ref:`hold_out_command-name` when no data is being held out (the old hold_out values were not erased). 06-14 ===== #. Improve the documentation for the :ref:`age_avg_table@Age Average Grid` . #. Group :ref:`option_table@Zero Sum Constraints` in option table documentation. #. Group :ref:`option_table@Extra Columns` in option table documentation. #. Group :ref:`option_table@Age Average Grid` in option table documentation. 06-13 ===== #. The :ref:`db2csv_command-name` would fail, if the previous fit had a :ref:`fit_command@warm_start` , with the following message: :: Previous fit command in log table used simulated data but cannot find data_sim table This has been fixed. #. Advance to cppad_mixed-20210613. Add the :ref:`mixed_info_table-name` table and the db2csv :ref:`db2csv_command@mixed_info.csv` output file. 06-12 ===== #. Add the :ref:`hold_out_command-name` . #. Change *out* column of the data.csv to include the result of all the hold_out settings. This has since been changed to the :ref:`db2csv_command@data.csv@d_out` and :ref:`db2csv_command@data.csv@s_out` columns. 06-11 ===== #. Add the :ref:`data_subset_table@hold_out` column to the data_subset table. #. Check that all the files output by the :ref:`init_command-name` are present when executing commands other than ``init`` and ``set`` . 06-10 ===== #. Add the :ref:`option_table@hold_out_integrand` option and :ref:`user_hold_out_1.py-name` example. #. Advance to cppad_mixed-20210610. This fixes a warning message during ``bin/get_cppad_mixed.sh`` on ubuntu. 06-09 ===== #. Remove ``_table`` from the links in :ref:`data_flow-name` where they are in lists of tables and it is not necessary. #. Change the ``fixed_trace`` table's name to ``trace_fixed`` . #. The :ref:`trace_fixed_table-name` and :ref:`fit_command@Output Tables@ipopt_info_table` were not being deleted by the :ref:`init_command-name` . This has been fixed. 06-08 ===== #. The :ref:`fit_command@warm_start` documentation was improved. #. Remove the tracing optimization and warn start wish list items because they have been completed. 06-07 ===== #. Advance to cppad_mixed-20210607 and add :ref:`trace_fixed_table@restoration` to the fixed_trace table. #. Add a discussion heading at the top of :ref:`database-name` sections that did not have a heading after the jump table at the top of the section; e.g., the covariate table :ref:`covariate_table@Discussion` heading. 06-06 ===== Advance to cppad_mixed-20210606. This enable adding the :ref:`fixed_trace_table` , for an example usage; see :ref:`user_warm_start.py@Fixed Trace Table` in the warm start example. 06-01 ===== The :ref:`fit_command@warm_start` option was added to the fit command. This is much better than just modifying the :ref:`start_var_table-name` because it also sets the initial Lagrange multipliers. For an example, see :ref:`user_warm_start.py-name` . 05-31 ===== Advance to cppad_mixed-20210531. This is in preparation for adding a warm start option to the fit command. 05-29 ===== #. Advance to cppad_mixed-20210528. This speeds up and improves the ``adaptive`` :ref:`option_table@Optimize Fixed and Random@derivative_test` . It also improves the checking of bounds, complementarity, and the gradient of the Lagrangian at the final solution. It also improves the detection of when Lagrangian multipliers should be non-zero because a constraint is active; e.g., see :ref:`fit_var_table@lagrange_value` in the fit_var table. #. The new version of cppad_mixed found identifiability problems with the :ref:`user_subgroup_mulcov.py-name` example. This has been fixed by change the prior for the child and subgroup random effect from uniform to a very broad Gaussian. 05-21 ===== #. Advance to :ref:`dock_dismod_at.sh-name` version 20210416. Building this new image will require executing {xrst_spell_off} {xrst_code sh} ./dock_dismod_at image mixed ./dock_dismod_at image dismod_at {xrst_code} {xrst_spell_on} The dismod_at.mixed image is necessary because the version of cppad_mixed has changed since the last advancement of dock_dismod_at.sh. #. In the ``user_data_sim_py`` example the code {xrst_spell_off} {xrst_code py} residual = (math.log(z + eta + delta) - math.log(mu + eta)) / sigma {xrst_code} {xrst_spell_on} was incorrect and was changed to {xrst_spell_off} {xrst_code py} residual = (math.log(z + eta) - math.log(mu + eta)) / sigma {xrst_code} {xrst_spell_on} #. Change the default setting for quasi_fixed from ``'true'`` to ``'false'`` . #. Change :ref:`user_fit_sim.py-name` *quasi_newton* setting from ``'true'`` to ``'false'`` and make the maximum allowable error smaller. 05-20 ===== {xrst_comment Merge simulate branch} During a :ref:`simulate_command-name` , the calculation of ``data_sim_delta`` and ``data_sim_stdcv`` in the :ref:`data_sim_table-name` often resulted in zero or negative values which does not make sense and would cause dismod_at to abort with the message ... ``simulate_command sim_stdcv is nan or less than zero`` ... This problem has been fixed using the adjusted standard deviation to replace *data_sim_delta* and the minimum CV standard deviation to replace *data_sim_stdcv* . These replacements use the original measurements :ref:`data_table@meas_value` instead of the simulated measurements :ref:`data_sim_table@data_sim_value` to define the corresponding transformed standard deviation sigma_i . #. Add the :ref:`user_sim_log.py-name` example to highlight this change. #. Adapt the :ref:`user_data_sim.py-name` examples to this change. #. **API Change** : The ``data_sim_delta`` and ``data_sim_stdcv`` columns were removed from the :ref:`data_sim_table-name` . The ``sim_stdcv`` and ``sim_delta`` columns were removed from the :ref:`db2csv_command@data.csv` . 05-19 ===== Add missing source code in the documentation for the :ref:`covid_19.py` and :ref:`residual.py` examples. 05-18 ===== Advance to cppad_mixed-20210517. This change, and changes to dismod_at, catch and report throws of ``std::exception`` both in the output and in the :ref:`log_table-name` . 05-15 ===== Advance to cppad_mixed-20210515. This avoids running the cppad_mixed tests during :ref:`example_install.sh-name` . It also fixes a problem with the optimization of cppad function objects. 05-07 ===== Advance to cppad_mixed-20210507. This should solve some install problems on the msys and cygwin systems. 05-06 ===== If a meas_delta , in the ``data.csv`` created by ``db2csv`` command, is greater than the maximum python ``float`` , report it as empty. 04-16 ===== #. The :ref:`fit both` command was failing when :ref:`option_table@Zero Sum Constraints@zero_sum_child_rate` was specified for a rate that did not have any children. This has been fixed. #. There was a syntax error in the ``test/user/fit_sim.py`` program. This has been fixed. #. Advance to :ref:`dock_dismod_at.sh-name` version 20210416. This includes the bug fix above. 03-02 ===== Advance to cppad_mixed-20210302. This places the release source directories Ipopt.git, Mumps.git, ASL.git in ``external/release/cppad_mixed/external`` (similarly for the debug case). Previously Mumps.git was below ASL.git and Ipopt.git was below Mumps.git. 01-28 ===== Advance to :ref:`dock_dismod_at.sh-name` version 20210128. Most importantly this includes the new *meas_noise_effect* options added on :ref:`2021@mm-dd@01-10` . For those at IHME, it is also has the new ``fit_ihme.py`` program. 01-22 ===== The :ref:`set` ``option`` command no longer requires the option table it to be consistent with all the other :ref:`input-name` tables. It only has to be self consistent. This enables one to use the ``set option`` command to change the :ref:`option_table@rate_case` to a value that is consistent with the rest of the database. 01-21 ===== The program ``db_simplify`` was changed to ``fit_ihme.py`` because it fits ihme databases. It now gets installed and is part of the User API. You can use it, if you have access to the ihme databases, but it is not part of this documentation (specifications). If you execute ``fit_ihme.py`` with no arguments, you will get a usage message that give you some idea how to use it. 01-19 ===== The file name was removed from the log table entry at the :ref:`python_main@Log Table@Start` of a ``dismodat.py`` command. This is so that all log messages with :ref:`log_table@message_type` ``command`` start with ``begin`` and then the name of the command. 01-11 ===== Change the order of the columns in the file :ref:`db2csv_command@variable.csv` so that the fit results :ref:`db2csv_command@variable.csv@fit_value` are next to the corresponding lower and upper limits. 01-10 ===== #. The :ref:`user_data_sim.py-name` example was changed so that it uses the same notation as the new version of the :ref:`data_sim_table-name` documentation and tests more cases. #. The bug fix of the value of *data_sim_stdcv* on 01-09 was corrected (better testing show it was incorrect). #. Add following :ref:`option_table@meas_noise_effect` cases: :ref:`data_like@Adjusted Standard Deviation, sigma_i(theta)@add_std_scale_none` , :ref:`data_like@Adjusted Standard Deviation, sigma_i(theta)@add_var_scale_none` . 01-09 ===== #. The documentation for the :ref:`data_sim_table-name` was improved. #. Fix a bug in the value of the data_sim table column *data_sim_stdcv* when :ref:`option_table@meas_noise_effect` is ``add_var_scale_log`` and the density is :ref:`density_table@Notation@Linear` . 01-08 ===== #. The definition of the transformed standard deviation :math:`\sigma` was extended to apply to all densities, not just :ref:`density_table@Notation@Log Scaled` densities. In addition, the definition was move from the :ref:`statistic-name` section to a place where the different cases are easier to understand. #. In the definition of the value prior for the random effects :ref:`random_value@V_j^u` the term .. math:: D \left( \theta_j \W{,} \mu_j \W{,} \lambda_j \sigma_j \W{,} d_j \right) was corrected to be .. math:: D \left( u_j \W{,} \mu_j \W{,} \lambda_j \sigma_j \W{,} d_j \right) {xrst_end 2021}