Ggthjy

what I am loading a yaml file and updating it with argparse. what is your feedback on how to update yaml parameters automatically without checking if a argparse parameter is None.
There are almost 20 configuration parameters that with this current version I have to put 20 if conditions.

the output of yaml.save_load is:

import os

import argparse

import yaml

import functools

import datetime



parser = argparse.ArgumentParser()

parser.add_argument(

    "--config",

    nargs="?",

    type=str,

    default="p.yml",

    help="Configuration file to use",

)

parser.add_argument('--learning_rate', type=float, default=0.001, help='learning rate amount')



args = parser.parse_args()





def yaml_dump(file_path, data):

    """

    Write data into yaml file in file_path

    :param file_path:

    :param data:

    :return: void

    """

    with open(file_path, 'w') as file_descriptor:

        yaml.safe_dump(data, file_descriptor)





def yaml_loader(file_path):

    """

    Load yaml file

    :param file_path:

    :return: yaml file configuration

    """

    with open(file_path, "r") as file_descriptor:

        return yaml.safe_load(file_descriptor)





cfg = yaml_loader(args.config)

if args.learning_rate is not None:

    cfg['training']['optimizer']['lr'] = args.learning_rate





ts = str(datetime.datetime.now()).split(".")[0].replace(" ", "_")

ts = ts.replace(":", "_").replace("-", "_")

logdir = os.path.join("runs", os.path.basename(args.config)[:-4], ts)

yaml_dump(os.path.join(logdir, args.config.split('/')[-1]), cfg)

here is the content of p.yaml file:

model:

    arch: fcn8s

data:

    dataset: pascal

    train_split: train

    val_split: val

    img_rows: 'same'

    img_cols: 'same'

    path: VOC/

    sbd_path: VOC/benchmark_RELEASE/

training:

    train_iters: 300000

    batch_size: 1

    val_interval: 1000

    n_workers: 16

    print_interval: 50

    optimizer:

        name: 'sgd'

        lr: 1.0e-10

        weight_decay: 0.0005

        momentum: 0.99

    loss:

        name: 'cross_entropy'

        size_average: False

    lr_schedule:

    resume: fcn8s_pascal_best_model.pkl

While you cannot iterate over a Namespace directly, you can parse it into a dictionary using vars and iterate over that, if you have a mapping from the argparse name to the path in the config file. The hardest part is using this path to get to the right level in the dictionary:

from argparse import Namespace



# Hard-code command line arguments instead of parsing them

args = Namespace(learning_rate=0.001)

# fake having read the config file

cfg = {"training": {"optimizer": {"lr": 0}}}

# mapping from argparse name to path in config file

cfg_path = {"learning_rate": ('training', 'optimizer', 'lr')}



for name, value in vars(args).items():

    if value is None:

        continue

    # go down to right level

    d = cfg

    keys = cfg_path[name]

    for key in keys[:-1]:

        d = d[key]

    # assign value, using the fact that dictionaries are mutable

    d[keys[-1]] = value

This will override values where you have a default value in argparse that is not None, which may for example change the order in which elements appear in your dictionaries.