Ggthjy

I'm working with this LSTM model

model = Sequential()

model.add(Masking(mask_value=0, input_shape=(timesteps, features)))

model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences=False))

model.add(Dense(features, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

and shapes X_train (21, 11, 5), y_train (21, 5).

Each timestep is represented by 5 features.
return_sequences is set to False because I want to predict one 5D array (the next timestep) for each input sequence of 11 timesteps.

I get the error
ValueError: Error when checking target: expected dense_1 to have 3 dimensions, but got array with shape (14, 5).

If I reshape the data as X_train (21, 11, 5), y_train (21, 1, 5) instead I get the errorValueError: Invalid shape for y: (14, 1, 5).

How should I solve this problem?

What are your features like? Given that you have a Dense layer outputting a softmax of size 5, this implies that all you want to predict is 1 feature, a categorical feature with 5 options.

If this is not true, more information about the features is needed to help here.

Your Y-variable for each feature should be of size (num_samples, time_step_len, num_categories_of_feature). You need to one-hot-encode each categorical feature separately, which gives the last dimension size, num_categories_of_feature. As you have it currently, the Y_train size is (num_samples, features). So, as you have the problem framed, the network has no way to learn the sequence patterns, as you only give it the end result. You should create your Y_train data to be the true value for the next time-step, for every time-step. Hence, (num_samples, time_step_len, num_categories_of_feature). Side note: I've only worked with LSTMs/RNN's on one problem, and this is how I did it. I cared about learning the sequences in it's entirety, because my inputs at prediction time are variable. If you always have 11 time-steps and always just want the next time-step prediction, this might not apply. I really don't know to be honest.

This is where I'm not totally sure if this is the only way to do this, but the way I think of this problem for wanting to predict 5 categorical variables, you need a way to output softmaxs for each variable. A softmax activation of size "features", like you have it here, is estimating a probability distribution of size 5, which implies your Y variable is only 1 categorical feature that has 5 potential values. So, you will need to set up your network to have 5 outputs with independent softmax outputs the size equal to the number of categories for each variable. A single softmax should only be used to estimate a distribution over a single class variable. 10 options for feat1? Softmax of size 10. etc.

losses = {"feat1_output": "categorical_crossentropy", "feat2_output": "categorical_crossentropy", "feat3_output": "categorical_crossentropy", "feat4_output": "categorical_crossentropy", "feat5_output": "categorical_crossentropy"}

lossWeights = {"feat1_output": 1.0, "feat2_output": 1.0, ... , ...}# if everything is equal, dont worry about specifying loss weights.

metrics = {"feat1_output": "categorical_accuracy", "feat2_output": "categorical_accuracy", "feat3_output": "categorical_accuracy", "feat4_output": "categorical_accuracy", "feat5_output": "categorical_accuracy"}

opt = Adam(lr=init_lr,decay=init_lr / num_epochs)

model.compile(loss = losses, loss_weights = lossWeights, optimizer=opt, metrics=metrics)

Now, you will be optimizing 5 loss functions at the same time, one for each categorical prediction. You must now have 5 Y-variable datasets, each of size (num_samples, time_step_len, num_categories_of_feature). You will then give 5 y datasets for the outputs in the fit function, as a list. However, to properly name the output layers, you will need to specify the names for the output layers in the model definition.