Making plots with the rail.plotting package
This notebook will show you the basics of making plots with the rail.plotting package
Setup and teardown scripts to setup a test area
[1]:
import os
from rail.projects import library
check_dir = os.path.basename(os.path.abspath(os.curdir))
if check_dir == 'examples':
os.chdir('..')
setup = library.setup_project_area()
assert setup == 0
# use this to cleanup
# library.teardown_project_area()
/home/docs/checkouts/readthedocs.org/user_builds/rail-projects/envs/latest/lib/python3.11/site-packages/ceci/__init__.py:12: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
from pkg_resources import DistributionNotFound
temp_data/
temp_data/data/
temp_data/data/ci_test_v1.1.3/
temp_data/data/ci_test_v1.1.3/9924/
temp_data/data/ci_test_v1.1.3/9924/part-0.parquet
temp_data/data/ci_test_v1.1.3/9925/
temp_data/data/ci_test_v1.1.3/9925/part-0.parquet
temp_data/projects/
temp_data/projects/ci_test/
temp_data/projects/ci_test/data/
temp_data/projects/ci_test/data/blend_baseline/
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin2.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin3_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_trainz.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin2_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin0_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin1_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin2_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin3.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_fzboost.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin4.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin1.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin4_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_simplenn.pkl
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin0_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin3.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin4_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin1_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin3_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin1_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin1_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_knn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin4_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin0_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_knn.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_trainz.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_gpz.pkl
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin1.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin2.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_gpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_fzboost.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_bpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin0.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin4_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin2_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_sompz.pkl
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin0.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin4.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_simplenn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin2_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin3_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_simplenn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin0_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin3_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_gpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_knn.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_fzboost.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_bpz.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_train_z.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_trainz.pkl
import the plotting control and load the objects from the yaml file
[2]:
from rail.plotting import control
[3]:
control.clear()
control.load_yaml('tests/ci_plot_groups.yaml')
plot_groups = control.get_plot_group_dict()
/home/docs/checkouts/readthedocs.org/user_builds/rail-projects/envs/latest/lib/python3.11/site-packages/qp/parameterizations/interp/interp.py:140: RuntimeWarning: There are non-finite values in the yvals for the following distributions: (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116,
117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129,
130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155,
156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168,
169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181,
182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259,
260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272,
273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298,
299, 300, 301]))
warnings.warn(
/home/docs/checkouts/readthedocs.org/user_builds/rail-projects/envs/latest/lib/python3.11/site-packages/qp/parameterizations/hist/hist.py:134: RuntimeWarning: There are non-finite values in the pdfs for the distributions: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0]
warnings.warn(
Print the stuff that we have loaded
[4]:
control.print_contents()
----------------
Plotter
zestimate_v_ztrue_hist2d: <class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueHist2D'>
zestimate_v_ztrue_profile: <class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueProfile'>
biweight_stats_v_redshift: <class 'rail.plotting.pz_plotters.PZPlotterBiweightStatsVsRedshift'>
biweight_stats_v_magnitude: <class 'rail.plotting.pz_plotters.PZPlotterBiweightStatsVsMag'>
accuracy_v_ztrue: <class 'rail.plotting.pz_plotters.PZPlotterAccuraciesVsTrue'>
tomo_bins: <class 'rail.plotting.nz_plotters.NZPlotterTomoBins'>
truth: <class 'rail.plotting.cat_plotters.CatPlotterTruth'>
magnitudes: <class 'rail.plotting.cat_plotters.CatPlotterMagntidues'>
magnitudes_v_truth: <class 'rail.plotting.cat_plotters.CatPlotterMagntiduesVsTruth'>
colors_v_truth: <class 'rail.plotting.cat_plotters.CatPlotterColorsVsTruth'>
----------------
PlotterList
zestimate_v_ztrue: ['zestimate_v_ztrue_hist2d', 'zestimate_v_ztrue_profile']
biweight_stats: ['biweight_stats_v_redshift', 'biweight_stats_v_magnitude']
accuracy_v_ztrue: ['accuracy_v_ztrue']
tomo_bins: ['tomo_bins']
catalog_plots: ['truth', 'magnitudes', 'magnitudes_v_truth', 'colors_v_truth']
----------------
----------------
Project
ci_test: tests/ci_project.yaml
----------------
Dataset
blend_baseline_trainz: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_trainz)
blend_baseline_knn: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_knn)
blend_baseline_fzboost: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_fzboost)
blend_baseline_all: RailPZMultiPointEstimateDataHolder( ['blend_baseline_trainz', 'blend_baseline_knn', 'blend_baseline_fzboost'] )
blend_baseline_tomo_knn_equal_count_naive_stack: RailNZTomoBinsDataHolder( ci_test, blend_baseline_knn_equal_count_naive_stack )
blend_baseline_truth_and_mags: RailCatTruthAndMagntiduesDataHolder ( ci_test, blend_baseline_test)
----------------
DatasetList
baseline_test: ['blend_baseline_trainz', 'blend_baseline_knn', 'blend_baseline_fzboost']
baseline_merged: ['blend_baseline_all']
blend_baseline_tomo_knn: ['blend_baseline_tomo_knn_equal_count_naive_stack']
blend_baseline_catalog: ['blend_baseline_truth_and_mags']
----------------
----------------
PlotGroup
zestimate_v_ztrue_test_plots: PlotGroup: zestimate_v_ztrue, DatasetList: baseline_test
accuracy_v_ztrue: PlotGroup: accuracy_v_ztrue, DatasetList: baseline_merged
biweight_stats: PlotGroup: biweight_stats, DatasetList: baseline_test
tomo_bins: PlotGroup: tomo_bins, DatasetList: blend_baseline_tomo_knn
catalog_plots: PlotGroup: catalog_plots, DatasetList: blend_baseline_catalog
----------------
[5]:
control.print_classes()
RailPlotter
RailPlotter <class 'rail.plotting.plotter.RailPlotter'>
PZPlotterPointEstimateVsTrueHist2D <class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueHist2D'>
PZPlotterPointEstimateVsTrueProfile <class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueProfile'>
PZPlotterAccuraciesVsTrue <class 'rail.plotting.pz_plotters.PZPlotterAccuraciesVsTrue'>
PZPlotterBiweightStatsVsRedshift <class 'rail.plotting.pz_plotters.PZPlotterBiweightStatsVsRedshift'>
PZPlotterBiweightStatsVsMag <class 'rail.plotting.pz_plotters.PZPlotterBiweightStatsVsMag'>
NZPlotterTomoBins <class 'rail.plotting.nz_plotters.NZPlotterTomoBins'>
CatPlotterTruth <class 'rail.plotting.cat_plotters.CatPlotterTruth'>
CatPlotterMagntidues <class 'rail.plotting.cat_plotters.CatPlotterMagntidues'>
CatPlotterMagntiduesVsTruth <class 'rail.plotting.cat_plotters.CatPlotterMagntiduesVsTruth'>
CatPlotterColorsVsTruth <class 'rail.plotting.cat_plotters.CatPlotterColorsVsTruth'>
----------------
RailDatasetHolder
RailDatasetHolder <class 'rail.plotting.dataset_holder.RailDatasetHolder'>
RailPZPointEstimateDataHolder <class 'rail.plotting.pz_data_holders.RailPZPointEstimateDataHolder'>
RailPZMultiPointEstimateDataHolder <class 'rail.plotting.pz_data_holders.RailPZMultiPointEstimateDataHolder'>
RailNZTomoBinsDataHolder <class 'rail.plotting.nz_data_holders.RailNZTomoBinsDataHolder'>
RailCatTruthAndMagntiduesDataHolder <class 'rail.plotting.cat_data_holders.RailCatTruthAndMagntiduesDataHolder'>
----------------
let’s have a look at the “PlotGroup” that we have loaded.
A “PlotGroup” points at one or more plots and one or more datasets and will iterate over both and make all the possible plots
[6]:
for k, v in plot_groups.items():
print(f"{k}: {v}")
zestimate_v_ztrue_test_plots: PlotGroup: zestimate_v_ztrue, DatasetList: baseline_test
accuracy_v_ztrue: PlotGroup: accuracy_v_ztrue, DatasetList: baseline_merged
biweight_stats: PlotGroup: biweight_stats, DatasetList: baseline_test
tomo_bins: PlotGroup: tomo_bins, DatasetList: blend_baseline_tomo_knn
catalog_plots: PlotGroup: catalog_plots, DatasetList: blend_baseline_catalog
Lets grab a particular plot_group
[7]:
plot_group = plot_groups['zestimate_v_ztrue_test_plots']
And let’s grab the associated datasets
[8]:
from rail.plotting.dataset_factory import RailDatasetFactory
dataset_factory = RailDatasetFactory.instance()
dataset_list = control.get_dataset_list(plot_group.config.dataset_list_name).resolve(dataset_factory)
[9]:
for k in dataset_list:
print(f"{k.config.name}: {k}")
blend_baseline_trainz: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_trainz)
blend_baseline_knn: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_knn)
blend_baseline_fzboost: RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_fzboost)
And lets get a particular dataset
[10]:
dataset_holder = dataset_list[0]
print(dataset_holder)
RailPZPointEstimateDataHolder ( ci_test, blend_baseline_test_trainz)
[11]:
dataset_holder.resolve()
[11]:
{'truth': array([0.81583488, 2.18756206, 0.66740374, ..., 0.63456433, 0.35929734,
2.95149095], shape=(100000,)),
'pointEstimate': array([0.49335548, 0.49335548, 0.49335548, ..., 0.49335548, 0.49335548,
0.49335548], shape=(100000,)),
'magnitude': array([24.41173884, 24.881559 , 21.3479835 , ..., 20.97517924,
21.43539084, 25.33788806], shape=(100000,))}
Now grab the associated plotters
[12]:
from rail.plotting.plotter_factory import RailPlotterFactory
plotter_factory = RailPlotterFactory.instance()
plotter_list = RailPlotterFactory.get_plotter_list(plot_group.config.plotter_list_name).resolve(plotter_factory)
[13]:
plotter_list
[13]:
[<class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueHist2D'>,
<class 'rail.plotting.pz_plotters.PZPlotterPointEstimateVsTrueProfile'>]
Get one plotter
[14]:
a_plotter = plotter_list[0]
Make a plot
[15]:
a_plotter.run(prefix="", **dataset_holder.resolve())
[15]:
{'zestimate_v_ztrue_hist2d': <rail.plotting.plot_holder.RailPlotHolder at 0x78ca56bc5350>}
Here we make all the define plots, this makes 6 plots ( 3 datasets X 2 plotters )
[16]:
from rail.plotting.plotter import RailPlotter
RailPlotter.iterate(plotter_list, dataset_list)
[16]:
{'blend_baseline_trainz': <rail.plotting.plot_holder.RailPlotDict at 0x78ca56e32050>,
'blend_baseline_knn': <rail.plotting.plot_holder.RailPlotDict at 0x78ca541860d0>,
'blend_baseline_fzboost': <rail.plotting.plot_holder.RailPlotDict at 0x78ca5419add0>}
[ ]:
[ ]: