DL之DNN:利用MultiLayerNetExtend模型【6*100+ReLU+SGD,dropout】对Mnist数据集训练来抑制过拟合
目录
190417更新
- class RMSprop:
- def __init__(self, lr=0.01, decay_rate = 0.99):
- self.lr = lr
- self.decay_rate = decay_rate
- self.h = None
-
- def update(self, params, grads):
- if self.h is None:
- self.h = {}
- for key, val in params.items():
- self.h[key] = np.zeros_like(val)
-
- for key in params.keys():
- self.h[key] *= self.decay_rate
- self.h[key] += (1 - self.decay_rate) * grads[key] * grads[key]
- params[key] -= self.lr * grads[key] / (np.sqrt(self.h[key]) + 1e-7)
-
-
- class Nesterov:
- def __init__(self, lr=0.01, momentum=0.9):
- self.lr = lr
- self.momentum = momentum
- self.v = None
-
- def update(self, params, grads):
- if self.v is None:
- self.v = {}
- for key, val in params.items():
- self.v[key] = np.zeros_like(val)
-
- for key in params.keys():
- self.v[key] *= self.momentum
- self.v[key] -= self.lr * grads[key]
- params[key] += self.momentum * self.momentum * self.v[key]
- params[key] -= (1 + self.momentum) * self.lr * grads[key]
-
-
- use_dropout = True
- dropout_ratio = 0.2
-
- network = MultiLayerNetExtend(input_size=784, hidden_size_list=[100, 100, 100, 100, 100, 100],
- output_size=10, use_dropout=use_dropout, dropout_ration=dropout_ratio)
- trainer = Trainer(network, x_train, t_train, x_test, t_test, epochs=301, mini_batch_size=100,
- optimizer='sgd', optimizer_param={'lr': 0.01}, verbose=True)
- trainer.train()
- train_acc_list, test_acc_list = trainer.train_acc_list, trainer.test_acc_list
-
1、DNN[6*100+ReLU,SGD]: accuracy of not dropout on Minist dataset
- train loss:2.3364575765992637
- === epoch:1, train acc:0.10333333333333333, test acc:0.1088 ===
- train loss:2.414526554119518
- train loss:2.341182306768928
- train loss:2.3072782723352496
- === epoch:2, train acc:0.09666666666666666, test acc:0.1103 ===
- train loss:2.2600377181768887
- train loss:2.263350960525319
- train loss:2.2708260374887645
-
- ……
-
- === epoch:298, train acc:1.0, test acc:0.7709 ===
- train loss:0.00755416896470134
- train loss:0.009934657874546435
- train loss:0.008421672959852643
- === epoch:299, train acc:1.0, test acc:0.7712 ===
- train loss:0.007142981215285884
- train loss:0.008205245499586114
- train loss:0.007319626293763803
- === epoch:300, train acc:1.0, test acc:0.7707 ===
- train loss:0.00752230499930163
- train loss:0.008431046288276818
- train loss:0.008067532729014863
- === epoch:301, train acc:1.0, test acc:0.7707 ===
- train loss:0.010729407851274233
- train loss:0.007776889701033221
- =============== Final Test Accuracy ===============
- test acc:0.771
2、DNN[6*100+ReLU,SGD]: accuracy of dropout(0.2) on Minist dataset
- train loss:2.3064018541384437
- === epoch:1, train acc:0.11, test acc:0.1112 ===
- train loss:2.316626942558816
- train loss:2.314434337198633
- train loss:2.318862771955365
- === epoch:2, train acc:0.11333333333333333, test acc:0.1128 ===
- train loss:2.3241989320140717
- train loss:2.317694982413387
- train loss:2.3079716553885006
-
- ……
-
- === epoch:298, train acc:0.6266666666666667, test acc:0.5168 ===
- train loss:1.2359381134877185
- train loss:1.2833380447791383
- train loss:1.2728131428100005
- === epoch:299, train acc:0.63, test acc:0.52 ===
- train loss:1.1687601000183936
- train loss:1.1435412548991142
- train loss:1.3854277174616834
- === epoch:300, train acc:0.6333333333333333, test acc:0.5244 ===
- train loss:1.3039470016588997
- train loss:1.2359979876607923
- train loss:1.2871396654831204
- === epoch:301, train acc:0.63, test acc:0.5257 ===
- train loss:1.1690084424502523
- train loss:1.1820777530873694
- =============== Final Test Accuracy ===============
- test acc:0.5269
相关文章
CSDN:2019.04.09起
网站声明:如果转载,请联系本站管理员。否则一切后果自行承担。
加入交流群
请使用微信扫一扫!