sagemaker-python-sdk: Hyperparameter values forcefully converted to strings, thus unable to pass a list

Good morning, @yangaws,

Yes, I am using my own container. No, I do not find it difficult to decode a string back to list… if I know that I have to do it.

The problem is that sagemaker changes the type of the passed value when it is not expected to do so. The sagemaker doc reads:

hyperparameters()
    Return the hyperparameters as a dictionary to use for training.
    
    The fit() method, which trains the model, calls this method to find the hyperparameters.

        Returns:        The hyperparameters.
        Return type:  dict[str, str]

Indeed, it mentions that the return type is dict[str, str]. However, this notice of the return type is present only for EstimatorBase.hyperparameters() and is absent for Estimator.hyperparameters().

Moreover, the Estimator.hyperparameters() method returns the dict with value types as they’ve been passed to Estimator such as lists, numbers, etc.

>>> hp = {'a': ['bb', 'cc'], 'x': 1.5, 125: False}
>>> estimator = sm.estimator.Estimator(image,
...                        role, 1, 'ml.c4.2xlarge',
...                        output_path="s3://{}/output".format(session.default_bucket()),
...                        sagemaker_session=session,
...                        hyperparameters=hp)
>>>
>>> estimator.hyperparameters()
{'a': ['bb', 'cc'], 'x': 1.5, 125: False}

So one is ok to assume that this is a “dictionary to use for training”.

Errors that can be introduced could be quite nasty. For example, the list ['bb', 'cc'] and the string "['bb', 'cc']" respond to the same calls, such as len() or __getitem__(), but with quite different results.

My understanding is that you’re being compliant to the following specification:

   "HyperParameters": { 
      "string" : "string" 
   }

(AWS Documentation - Amazon SageMaker - Developer Guide - API Reference - Actions - Amazon SageMaker Service - CreateTrainingJob)

Forcing hyperparameter values to be strings looks rather unwarranted. I am interested what is the reason behind it. To work around, I have to implement additional checks to differentiate between reading “conventional” json and “sagemaker-style” json, doing additional decoding for the latter. I see this as an unnecessary complication.

If this restriction cannot be lifted, I suggest that

• the sagemaker doc explicitly and consistently states that all hyperparameter values (as well as keys) will be converted to strings;
• Estimator.hyperparameters() returns the dictionary with keys and values converted to strings (as it’s probably supposed to do).

Hey!

This is something that has being a pain for us too.

Do you find decoding the hyperparameter from string to list hard? If so, what’s the problem?

Yes. The right solution to adapt ourselves to sagemaker’s contract with hyperparameters is: keeping track of all possible key/values and their types in a custom image, just to parse it back to their expected format. And undoing the str() applied by sagemaker, casting every hyperparameter that is not a string is quite time consuming.

This is not easy to do when we have to send dozens of hyperparameters. Also, it is quite strange to send something like

{
    "objective": "binary",
    "boosting_type": "gbdt",
    "num_leaves": 31,
    "metric": ["binary_logloss", "auc"],
    "seed": None
} 

and getting inside of the container something like:

{
    "objective": "binary",
    "boosting_type": "gbdt",
    "num_leaves": "31",
    "metric": "[\"binary_logloss\", \"auc\"]",
    "seed": "None"
} 

Is there any chance of changing how sagemaker deals with hyperparameters?

I imagine that the decision of forcing an str() conversion was taken to prevent having to deal with json encoders or it’s more related to some concern with safety. Therefore, letting us control the json encoding and decoding and expecting just a json string as an argument of Estimator would help both sides?

Nothing? I am currently trying to pass a list of str (or int even) to an estimator and getting my list cut at the comma. Any developments on this issue?

Hi team, for those still having trouble here, I’ve been using these two functions to parse out the hyperparameters in my own estimator containers (it also strips leading 0’s, just in case you work with coordinates in your day to day, which can cause trouble when trying to cast to an int or float):

def destringify_dict_values(d):
    return {k: destringify(v) for k, v in d.items()}

def destringify(s):

    if s == len(s) * "0":
        return 0
    else:
        s = s.lstrip("0")

    if isinstance(s, str):
        try:
            val = literal_eval(s)
        except ValueError:
            val = s
    else:
        val = s

    if isinstance(val, float):
        if val.is_integer():
            return int(val)
        return val

    return val

It uses literal_eval from ast, which is not ideal, but still much safer than eval , since it raises a ValueError if it can’t cast to a basic Python type.

any updates? parsing values can be a pain if it’s all nested and has different data types. This adds a lot of extra workload. If user simply passes in a python dict, they would expect to get the same thru a simple load using the json module.

Hi

Currently I’m doing

https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-train-model.html

I’m in: “Step 4: Train a Model”

And in the “3. Set the hyperparameters for the XGBoost algorithm by calling the set_hyperparameters method of the estimator.”

I’m having a problem after executing: xgb_model.fit({"train": train_input, "validation": validation_input}, wait=True)

After going through some documentation: https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/

https://xgboost.readthedocs.io/en/latest/parameter.html

I see that the list of available options in objective

This is how I have it right now 1:

xgb_model.set_hyperparameters(
    max_depth = 5,
    eta = 0.2,
    gamma = 4,
    min_child_weight = 6,
    subsample = 0.7,
    objective = "binary:logistic",
    num_round = 1000
)

I’ve tried to use

2:

...
 objective = {"binary": "logistic"},
 ...

3:

...
 objective = ['binary: logistic'],
 ...

The error output varies but it’s always like this:

1. Failed to parse hyperparameter objective value binary:logistic to Json.

hyperparameter_validation.py", line 47, in validate_range raise exc.UserError(“Hyperparameter {}: {} is not in {}”.format(self.name, value, self.range)) sagemaker_algorithm_toolkit.exceptions.UserError: Hyperparameter objective: {‘binary:logistic’} is not in [‘aft_loss_distribution’, ‘binary:logistic’, ‘binary:logitraw’, ‘binary:hinge’, ‘count:poisson’, ‘multi:softmax’, ‘multi:softprob’, ‘rank:pairwise’, ‘rank:ndcg’, ‘rank:map’, ‘reg:linear’, ‘reg:squarederror’, ‘reg:logistic’, ‘reg:gamma’, ‘reg:pseudohubererror’, ‘reg:squaredlogerror’, ‘reg:tweedie’, ‘survival:aft’, ‘survival:cox’]

Do you guys now if I’m doing something wrong? I would think the tutorial is a no brainer but maybe I’m missing something

we are facing this issue as well. And in our case it is nested dict (we use our customer docker images). its very hard to reparse this to json because we have different types of values.

you guys also convert double quotes to single quotes, so to effectively get this to work, i have to do this

param_grid = {
            "svd__n_components": [ json.loads(training_params["svd"].replace("\'", "\""))["n_components"]],
            "xgb__eval_set": [(X_test, y_test)],
        }

Glad to create a PR for this immediately. I can’t believe this has not been addressed in nearly a year.

+1 I wasted a whole day of my life trying to debug this…training the docker container worked locally, but on sagemaker it did not! Truly an insidious issue.

What is the status on this issue? I believe the relevant changes were not included on v2.0.0. @laurenyu