Using the `rail.projects` analysis component library

This notebook will show you the basics of using the rail.projects.library component library

Setup and teardown scripts to setup a test area

[1]:

import os
from rail.projects import library

check_dir = os.path.basename(os.path.abspath(os.curdir))
if check_dir == 'examples':
    os.chdir('..')

setup = library.setup_project_area()
assert setup == 0

# use this to cleanup
# library.teardown_project_area()

/home/docs/checkouts/readthedocs.org/user_builds/rail-projects/envs/stable/lib/python3.11/site-packages/ceci/__init__.py:12: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import DistributionNotFound

temp_data/
temp_data/data/
temp_data/data/ci_test_v1.1.3/
temp_data/data/ci_test_v1.1.3/9924/
temp_data/data/ci_test_v1.1.3/9924/part-0.parquet
temp_data/data/ci_test_v1.1.3/9925/
temp_data/data/ci_test_v1.1.3/9925/part-0.parquet
temp_data/projects/
temp_data/projects/ci_test/
temp_data/projects/ci_test/data/
temp_data/projects/ci_test/data/blend_baseline/
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin2.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin3_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_trainz.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin2_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin0_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin1_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin2_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin3.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_fzboost.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin4.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin1.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin4_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_simplenn.pkl
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin0_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin3.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin4_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin1_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin3_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin1_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin1_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_knn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin4_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin0_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_knn.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_trainz.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_gpz.pkl
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin1.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin2.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_gpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_fzboost.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_bpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_uniform_binning_bin0.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin4_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin2_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_sompz.pkl
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin0.hdf5
temp_data/projects/ci_test/data/blend_baseline/true_NZ_true_nz_knn_equal_count_bin4.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_estimate_simplenn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin2_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_equal_count_bin3_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_simplenn.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin0_naive_stack.hdf5
temp_data/projects/ci_test/data/blend_baseline/single_NZ_summarize_knn_uniform_binning_bin3_point_est_hist.hdf5
temp_data/projects/ci_test/data/blend_baseline/output_evaluate_gpz.hdf5
temp_data/projects/ci_test/data/blend_baseline/model_inform_knn.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_fzboost.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_bpz.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_train_z.pkl
temp_data/projects/ci_test/data/blend_baseline/model_inform_trainz.pkl

[2]:

# Import the project library code
from rail.projects import library

Loading a library from a yaml file

This will read in a yaml file, and contstuct objects and templates that are described in the yaml file

[3]:

# Load a particular library
library.load_yaml('tests/ci_project_library.yaml')

[4]:

# Print the contents of the loaded library
library.print_contents()

----------------
Algorithms
----------------
PZAlgorithm
  trainz: rail.estimation.algos.train_z.{'Estimate': 'TrainZEstimator', 'Inform': 'TrainZInformer'}
  simplenn: rail.estimation.algos.sklearn_neurnet.{'Estimate': 'SklNeurNetEstimator', 'Inform': 'SklNeurNetInformer'}
  fzboost: rail.estimation.algos.flexzboost.{'Estimate': 'FlexZBoostEstimator', 'Inform': 'FlexZBoostInformer'}
  knn: rail.estimation.algos.k_nearneigh.{'Estimate': 'KNearNeighEstimator', 'Inform': 'KNearNeighInformer'}
  gpz: rail.estimation.algos.gpz.{'Estimate': 'GPzEstimator', 'Inform': 'GPzInformer'}
----------------
SpecSelection
  zCOSMOS: rail.creation.degraders.spectroscopic_selections.{'Select': 'SpecSelection_zCOSMOS'}
----------------
Classifier
  equal_count: rail.estimation.algos.equal_count.{'Classify': 'EqualCountClassifier'}
  uniform_binning: rail.estimation.algos.uniform_binning.{'Classify': 'UniformBinningClassifier'}
----------------
Summarizer
  naive_stack: rail.estimation.algos.naive_stack.{'Summarize': 'NaiveStackMaskedSummarizer'}
  point_est_hist: rail.estimation.algos.point_est_hist.{'Summarize': 'PointEstHistMaskedSummarizer'}
----------------
ErrorModel
  lsst: rail.creation.degraders.photometric_errors.{'ErrorModel': 'LSSTErrorModel', 'Bands': ['u', 'g', 'r', 'i', 'z', 'y']}
  roman: rail.creation.degraders.photometric_errors.{'ErrorModel': 'RomanErrorModel', 'Bands': ['u', 'g', 'r', 'i', 'z', 'y']}
----------------
Subsampler
  random_subsampler: rail.projects.subsampler.{'Subsample': 'RandomSubsampler'}
----------------
Reducer
  roman_rubin: rail.projects.reducer.{'Reduce': 'RomanRubinReducer'}
----------------

----------------
Catalogs
----------------
CatalogTemplate
  truth: {catalogs_dir}/{project}_{sim_version}/{healpix}/part-0.parquet
  reduced: {catalogs_dir}/{project}_{sim_version}_{selection}/{healpix}/part-0.pq
  degraded: {catalogs_dir}/{project}_{sim_version}_{selection}_{flavor}/{healpix}/{basename}
----------------
CatalogInstance
----------------

----------------
Pipelines:
----------------
PipelineTemplate
  truth_to_observed: rail.pipelines.degradation.truth_to_observed.TruthToObservedPipeline
  prepare: rail.pipelines.utils.prepare_observed.PrepareObservedPipeline
  photometric_errors: rail.pipelines.degradation.apply_phot_errors.ApplyPhotErrorsPipeline
  blending: rail.pipelines.degradation.blending.BlendingPipeline
  spec_selection: rail.pipelines.degradation.spectroscopic_selection_pipeline.SpectroscopicSelectionPipeline
  inform: rail.pipelines.estimation.inform_all.InformPipeline
  estimate: rail.pipelines.estimation.estimate_all.EstimatePipeline
  evaluate: rail.pipelines.evaluation.evaluate_all.EvaluationPipeline
  pz: rail.pipelines.estimation.pz_all.PzPipeline
  inform_sompz: rail.pipelines.estimation.inform_sompz.InformSomPZPipeline
  estimate_sompz: rail.pipelines.estimation.estimate_sompz.EstimateSomPZPipeline
  inform_somlike: rail.pipelines.calib.inform_somlike.InformSomlikePipeline
  somlike_recalib: rail.pipelines.calib.somlike_recalib.SomlikeRecalibPipeline
  inform_recalib: rail.pipelines.calib.inform_recalib.InformRecalibPipeline
  estimate_recalib: rail.pipelines.calib.estimate_recalib.EstimateRecalibPipeline
  tomography: rail.pipelines.estimation.tomography.TomographyPipeline
----------------
PipelineInstance
----------------

----------------
Files:
----------------
FileTemplate
  test_file_100k: test_file_100k
  train_file_100k: train_file_100k
  train_file_zCOSMOS_100k: train_file_zCOSMOS_100k
  train_file_10: train_file_10
  deep_file_full: deep_file_full
  deep_file_small: deep_file_small
  wide_file_full: wide_file_full
  wide_file_small: wide_file_small
  spec_file_full: spec_file_full
  spec_file_small: spec_file_small
----------------
FileInstance
----------------

----------------
Selection
  maglim_25.5: cuts={'maglim_i': [None, 25.5]}
  gold: cuts={'maglim_i': [None, 25.5]}
  blend: cuts={'maglim_i': [None, 26.0]}
  crap: cuts={'maglim_i': [None, 30.0]}
  all: cuts={'maglim_i': [None, None]}
----------------

----------------
Subsample
  test_100k: N=100000 seed=1234
  train_100k: N=100000 seed=4321
  train_10: N=100000 seed=5555
----------------

Catalogs, Templates and Interpolats

Let’s get a particular “CatalogTemplate”, i.e, the one for the “truth” catalog

[5]:

a_catalog_template = library.get_catalog_template('truth')

Let’s get some information about this template

[6]:

print(f"{type(a_catalog_template)} called '{a_catalog_template.config.name}'")
print(f"has path '{a_catalog_template}' and iterates over {a_catalog_template.config.iteration_vars}")

<class 'rail.projects.catalog_template.RailProjectCatalogTemplate'> called 'truth'
has path '{catalogs_dir}/{project}_{sim_version}/{healpix}/part-0.parquet' and iterates over ['healpix']

[7]:

#dir(a_catalog_template)
#help(a_catalog_template)

Let’s use this template to make a particular version of the catalog,

[8]:

a_catalog_instance = a_catalog_template.make_catalog_instance(
    "truth_cat", catalogs_dir='tests/temp_dir', project='ci_test', sim_version='test',
)

Let’s get some information about this catalog

[9]:

print(f"{type(a_catalog_instance)} called '{a_catalog_instance.config.name}'")
print(f"has path '{a_catalog_instance}' and iterates over {a_catalog_instance.config.iteration_vars}")

<class 'rail.projects.catalog_template.RailProjectCatalogInstance'> called 'truth_cat'
has path 'tests/temp_dir/ci_test_test/{healpix}/part-0.parquet' and iterates over ['healpix']

Let’s get the files associated to this catalog for some particular pixels

[10]:

a_catalog_instance.resolve(healpix=[5055, 5051])

[10]:

['tests/temp_dir/ci_test_test/5055/part-0.parquet',
 'tests/temp_dir/ci_test_test/5051/part-0.parquet']

Algorithm holders and algorithms

Let’s get the object that hold information about a particular algorithn

[11]:

lsst_error_model_holder = library.get_algorithm('ErrorModel', 'lsst')

Let’s get some information about this algorithm

[12]:

print(f"{type(lsst_error_model_holder)} called '{lsst_error_model_holder.config.name}'")
print(f"has classes '{lsst_error_model_holder}'")

<class 'rail.projects.algorithm_holder.RailErrorModelAlgorithmHolder'> called 'lsst'
has classes 'rail.creation.degraders.photometric_errors.{'ErrorModel': 'LSSTErrorModel', 'Bands': ['u', 'g', 'r', 'i', 'z', 'y']}'

[13]:

lsst_error_model_class = library.get_algorithm_class('ErrorModel', 'lsst', 'ErrorModel')

[14]:

lsst_error_model_class.config_options

[14]:

{'output_mode': Parameter(What to do with the outputs, type: <class 'str'>, default: default [optional]),
 'seed': Parameter(Set to an `int` to force reproducible results., type: None, default: None [optional])}

[ ]:

[15]:

knn_holder = library.get_algorithm('PZAlgorithm', 'knn')

[16]:

print(f"{type(knn_holder)} called '{knn_holder.config.name}'")
print(f"has classes '{knn_holder}'")

<class 'rail.projects.algorithm_holder.RailPZAlgorithmHolder'> called 'knn'
has classes 'rail.estimation.algos.k_nearneigh.{'Estimate': 'KNearNeighEstimator', 'Inform': 'KNearNeighInformer'}'

[17]:

knn_inform_class = knn_holder.resolve('Inform')
knn_estimate_class = knn_holder.resolve('Estimate')
print(f"{knn_inform_class} {knn_estimate_class}")

<class 'rail.estimation.algos.k_nearneigh.KNearNeighInformer'> <class 'rail.estimation.algos.k_nearneigh.KNearNeighEstimator'>

[ ]:

Named subsamples

Let’s get the object that holds information about a named subsample of data

[18]:

subsample = library.get_subsample('test_100k')

[19]:

print(f"{type(subsample)} called '{subsample.config.name}'")
print(f"has options '{subsample}'")

<class 'rail.projects.subsample_factory.RailSubsample'> called 'test_100k'
has options 'N=100000 seed=1234'

[ ]:

Named selections

Let’s get the object that holds information about a named selections of data

[20]:

selection = library.get_selection('gold')

[21]:

print(f"{type(selection)} called '{selection.config.name}'")
print(f"has {selection}")

<class 'rail.projects.selection_factory.RailSelection'> called 'gold'
has cuts={'maglim_i': [None, 25.5]}

[ ]:

[ ]:

Using the rail.projects analysis component library