一站式服务理念,建设部中国建造师网查询,中国域名拍卖网,电商网站程序在本文中#xff0c;我已经实现了具有Dropout和L2正则化的人工神经网络的完全向量化代码。在本文中#xff0c;我实现了一个在多个数据集上测试的人工神经网络的完全向量化python代码。此外#xff0c;并对Dropout和L2正则化技术进行了实现和详细说明。强烈建议通过人工神经…在本文中我已经实现了具有Dropout和L2正则化的人工神经网络的完全向量化代码。 在本文中我实现了一个在多个数据集上测试的人工神经网络的完全向量化python代码。此外并对Dropout和L2正则化技术进行了实现和详细说明。强烈建议通过人工神经网络的基本工作正向传播和反向传播。本文分为10个部分介绍先决条件导入我们的库激活函数及求导神经网络类初始化权重和偏差正向传播成本函数反向传播预测新数据集的标签1.简介人工神经网络是监督深度学习中最简单和最基本的概念之一。它可用于执行多个任务如二进制或分类。它看起来很容易理解和实现。在编写这样一个网络的过程中会出现一些小问题这些问题会导致很大的错误并帮助您理解以前忽略的概念。因此在本文中我尝试实现一个人工神经网络它可能会帮助您节省正确编码和理解主题的每个概念所需的工作时间。2.先决条件我假设你知道神经网络是什么以及它们是如何学习的。如果你对Python和像numpy这样的库很熟悉的话这将很容易理解。另外还需要对线性代数和微积分知识有很好的了解以便于轻松地理解正向和反向传播部分。此外我强烈建议您阅读Andrew Ng在Coursera上的课程视频(https://www.coursera.org/ https://www.deeplearning.ai/ )。3.导入我们的库现在我们可以开始对神经网络进行编码了。第一件事是需要导入我们实现网络所需的所有库。# Importing the librariesimport numpy as npimport matplotlib.pyplot as pltimport pandas as pdimport warningsimport timewarnings.filterwarnings(ignore);import osimport sys我们将使用pandas导入和清理我们的数据集。Numpy 是执行矩阵代数和复杂计算的最重要的库。4.激活函数及求导在本文后面的部分中我们将需要激活函数来执行正向传播。另外我们还需要在反向传播过程中对激活函数求导。所以让我们编写一些激活函数。def sigmoid(z) : Reutrns the element wise sigmoid function. return 1./(1 np.exp(-z))def sigmoid_prime(z) : Returns the derivative of the sigmoid function. return sigmoid(z)*(1-sigmoid(z))def ReLU(z) : Reutrns the element wise ReLU function. return (z*(z 0))def ReLU_prime(z) : Returns the derivative of the ReLU function. return 1*(z0)def lReLU(z) : Reutrns the element wise leaky ReLU function. return np.maximum(z/100,z)def lReLU_prime(z) : Returns the derivative of the leaky ReLU function. z 1*(z0) z[z0] 1/100 return zdef tanh(z) : Reutrns the element wise hyperbolic tangent function. return np.tanh(z)def tanh_prime(z) : Returns the derivative of the tanh function. return (1-tanh(z)**2)# A dictionary of our activation functionsPHI {sigmoid:sigmoid, relu:ReLU, lrelu:lReLU, tanh:tanh}# A dictionary containing the derivatives of our activation functionsPHI_PRIME {sigmoid:sigmoid_prime, relu:ReLU_prime, lrelu:lReLU_prime, tanh:tanh_prime}我们有四种最流行的激活函数。首先是常规的sigmoid激活函数。我们有ReLU或“ 线性整流函数 ”。我们将主要使用这个激活函数。注意我们将保持ReLU 0的导数在点0处。我们还有一个ReLU的扩展版本叫做Leaky ReLU。它的工作原理与ReLU类似可以在某些数据集上提供更好的结果(不一定是全部)。我们还有tanh(双曲正切)激活函数。它也被广泛使用并且几乎总是优于sigmoid。另外PHI和PHI_PRIME是分别包含激活函数及其导数的python字典。5.我们的神经网络类在本节中我们将创建并初始化我们的神经网络类。首先我们将决定在初始化期间使用哪些参数。我们需要每层神经元的数量我们想在每一层中使用激活函数我们的特征矩阵(X包含行特征和列特征)。特征矩阵对应的标签(y是行向量)初始化权重和偏差的方法使用损失函数记住这一点让我们开始编写神经网络的类class NeuralNet : This is a class for making Artificial Neural Networks. L2 and Droupout are the default regularization methods implemented in this class. It takes the following parameters: 1. layers : A python list containing the different number of neurons in each layer. (containing the output layer) Eg - [64,32,16,16,1] 2. X : Matrix of features with rows as features and columns as different examples. 3. y : Numpy array containing the ouputs of coresponding examples. 4. ac_funcs : A python list containing activation function of each layer. Eg - [relu,relu,lrelu,tanh,sigmoid] 5. init_method : Meathod to initialize weights of the network. Can be gaussian,random,zeros. 6. loss_func : Currently not implemented 7. W : Weights of a pretrained neural network with same architecture. 8. W : Biases of a pretrained neural network with same architecture. 现在我们有了一个正确记录的神经网络类我们可以继续初始化网络的其他变量。 def __init__(self, layers, X, y, ac_funcs, init_methodgaussian, loss_funcb_ce, Wnp.array([]), Bnp.array([])) : Initialize the network. # Store the layers of the network self.layers layers # ---- self.W None self.B None # Store the number of examples in the dataset as m self.m X.shape[1] # Store the full layer list as n self.n [X.shape[0], *layers] # Save the dataset self.X X # Save coresponding output self.y y # List to store the cost of the model calculated during training self.cost [] # Stores the accuracy obtained on the test set. self.acc 0 # Activation function of each layer self.ac_funcs ac_funcs self.loss loss_func # Inittialize the weights by provided methods if not provided.我们将使用 self.m来存储数据集中示例的数量。 self.n 将存储每层中神经元数量的信息。 self.ac_funcs 是每层的激活函数的python列表。 self.cost 将在我们训练网络时存储成本函数的记录值。 self.acc 将在训练后的数据集上存储记录的精度。在初始化网络的所有变量之后让我们进一步初始化网络的权重和偏差。6.初始化权重和偏差我们知道权重不能初始化为零因为每个神经元的假设变得相同而网络永远不会学习。因此我们必须有一些方法来使我们的神经网络学习。我们可以使用高斯正态分布来获得随机值。由于这些分布的均值为零因此权重集中在零并且非常小。因此网络开始非常快速有效地学习。我们可以使用np.random.randn()函数从正态分布中生成随机值。 # Inittialize the weights by provided methods if not provided. if len(W) and len(B) : self.W W self.B B else : if init_methodgaussian: self.W [np.random.randn(self.n[nl], self.n[nl-1]) for nl in range(1,len(self.n))] self.B [np.zeros((nl,1), float32) for nl in self.layers] elif init_method random: self.W [np.random.rand(self.n[nl], self.n[nl-1]) for nl in range(1,len(self.n))] self.B [np.random.rand(nl,1) for nl in self.layers] elif init_method zeros: self.W [np.zeros((self.n[nl], self.n[nl-1]), float32) for nl in range(1,len(self.n))] self.B [np.zeros((nl,1), float32) for nl in self.layers]我们已将权重初始化为正态分布中的随机值。偏差已初始化为零。7.正向传播首先让我们理解没有正则化的正向传播。我们用Z表示每个神经元从一层到另一层的连接。一旦我们计算了Z我们将激活函数f应用于Z值以获得每层中每个神经元的激活y。这是简单的正向传播。Dropout是一种提高神经网络泛化能力和鲁棒性的神奇技术。所以让我们首先了解一下Dropout正则化。Dropout正则化的本质Dropout顾名思义是指神经网络中的一些神经元“失活”并对其余的神经元进行训练。为了提高性能我们可以训练几十个和几百个具有不同超参数值的神经网络获得所有网络的输出并取其平均值来获得最终结果。这个过程在计算上非常昂贵并且实际上不能实现。因此我们需要一种以更优化和计算成本更低的方式执行类似操作的方法。Dropout正则化以非常便宜和简单的方式完成相似的事情。事实上Dropout是优化性能的简单方法近年来受到了广泛的关注并且几乎在许多其他深度学习模型中无处不在。要实现Dropout我们将使用以下方法我们将首先从伯努利分布中提取随机值如果概率高于某个阈值则保持神经元不变然后执行常规正向传播。注意我们不会在预测新数据集上的值或测试时间期间应用dropout。实现Dropout的代码我们使用keep_prob作为每层神经元存活的概率。我们只保留概率高于存活概率或keep_prob的神经元。假设它的值是0.8。这意味着我们将使每一层中20%的神经元失活并训练其余80%的神经元。注意我们在每次迭代后停用随机选择的神经元。这有助于神经元学习在更大的数据集上泛化的特征。 def _feedForward(self, keep_prob): Forward pass z [];a [] z.append(np.dot(self.W[0], self.X) self.B[0]) a.append(PHI[self.ac_funcs[0]](z[-1])) for l in range(1,len(self.layers)): z.append(np.dot(self.W[l], a[-1]) self.B[l]) # a.append(PHI[self.ac_funcs[l]](z[l])) _a PHI[self.ac_funcs[l]](z[l]) a.append( ((np.random.rand(_a.shape[0],1) keep_prob)*_a)/keep_prob ) return z,a我们首先初始化将存储Z和A值的列表。我们首先在Z中附加第一层的线性值然后在A中附加第一层神经元的激活。PHI是一个python字典包含我们之前编写的激活函数。类似地使用for循环计算所有其他层的Z和A的值。注意我们没有在输入层应用dropout。我们最终返回Z和A的计算值。8.成本函数我们将使用标准二进制/分类交叉熵成本函数。 def _cost_func(self, a, _lambda): Binary Cross Entropy Cost Function return ( (-1/self.m)*np.sum(np.nan_to_num(self.y*np.log(a) (1-self.y)*np.log(1-a))) (_lambda/(2*self.m))*np.sum([np.sum(i**2) for i in self.W]) ) def _cost_derivative(self, a) : The derivative of cost w.r.t z return (a-self.y) 我们用L2正则化对我们的成本函数进行了编译。lambda参数称为“ 惩罚参数 ”。它有助于使权重值不会迅速增加从而更好地形成。这里 a包含输出层的激活值。我们还有函数_cost_derivative来计算成本函数对输出层激活的导数。我们稍后会在反向传播期间使用它。9.反向传播以下是我们需要执行反向传播的一些公式。我们将在深度神经网络上实现这一点。右边的公式是完全向量化的。一旦理解了这些公式我们就可以继续对它们进行编译。 def startTraining(self, epochs, alpha, _lambda, keep_prob0.5, interval100): Start training the neural network. It takes the followng parameters : 1. epochs : Number of epochs for which you want to train the network. 2. alpha : The learning rate of your network. 3. _lambda : L2 regularization parameter or the penalization parameter. 4. keep_prob : Dropout regularization parameter. The probability of neurons to deactivate. Eg - 0.8 means 20% of the neurons have been deactivated. 5. interval : The interval between updates of cost and accuracy. start time.time() for i in range(epochs1) : z,a self._feedForward(keep_prob) delta self._cost_derivative(a[-1]) for l in range(1,len(z)) : delta_w np.dot(delta, a[-l-1].T) (_lambda)*self.W[-l] delta_b np.sum(delta, axis1, keepdimsTrue) delta np.dot(self.W[-l].T, delta)*PHI_PRIME[self.ac_funcs[-l-1]](z[-l-1]) self.W[-l] self.W[-l] - (alpha/self.m)*delta_w self.B[-l] self.B[-l] - (alpha/self.m)*delta_b delta_w np.dot(delta, self.X.T ) (_lambda)*self.W[0] delta_b np.sum(delta, axis1, keepdimsTrue) self.W[0] self.W[0] - (alpha/self.m)*delta_w self.B[0] self.B[0] - (alpha/self.m)*delta_b我们将epochs、alpha(学习率)、_lambda、keep_prob和interval作为函数的参数来实现反向传播。我们从正向传播开始。然后我们将成本函数的导数计算为delta。现在对于每一层我们计算delta_w和delta_b其中包含成本函数对网络的权重和偏差的导数。然后我们根据各自的公式更新delta权重和偏差。在将最后一层的权重和偏差更新到第二层之后我们更新第一层的权重和偏差。我们这样做了几次迭代直到权重和偏差的值收敛。重要提示此处可能出现的一个重大错误是在更新权重和偏差后更新delta 。这样做可能会导致非常糟糕的梯度渐变消失/爆炸问题。我们的大部分工作都在这里完成但是我们仍然需要编译可以预测新数据集结果的函数。因此作为我们的最后一步我们将编写一个函数来预测新数据集的标签。10.预测新数据集的标签这一步非常简单。我们只需要执行正向传播但不需要Dropout正则化。我们不需要在测试时应用Dropout正则化因为我们需要所有层的所有神经元来为我们提供适当的结果而不仅仅是一些随机值。 def predict(self, X_test) : Predict the labels for a new dataset. Returns probability. a PHI[self.ac_funcs[0]](np.dot(self.W[0], X_test) self.B[0]) for l in range(1,len(self.layers)): a PHI[self.ac_funcs[l]](np.dot(self.W[l], a) self.B[l]) return a我们将返回输出层的激活作为结果。整个代码以下是您自己实现人工神经网络的完整代码。我在培训时添加了一些代码来打印网络的成本和准确性。除此之外一切都是一样的。# Importing the librariesimport numpy as npimport matplotlib.pyplot as pltimport pandas as pdimport warningsimport timewarnings.filterwarnings(ignore);import osimport sys# Importing our datasetos.chdir(C:/Users/Hilak/Desktop/INTERESTS/Machine Learning A-Z Template Folder/Part 3 - Classification/Section 14 - Logistic Regression);training_set pd.read_csv(Social_Network_Ads.csv);# Splitting our dataset into matrix of features and output values.X training_set.iloc[:, 1:4].valuesy training_set.iloc[:, 4].values# Encoding our object features.from sklearn.preprocessing import LabelEncoder, OneHotEncoderle_x LabelEncoder()X[:,0] le_x.fit_transform(X[:,0])ohe OneHotEncoder(categorical_features [0])X ohe.fit_transform(X).toarray()# Performing Feature scalingfrom sklearn.preprocessing import StandardScalerss StandardScaler()X[:,2:4] ss.fit_transform(X[:, 2:4])# Splitting the dataset into train and test set.from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test train_test_split(X,y,test_size0.3)X_train X_train.TX_test X_test.T# # Alternate Dataset for test purposes. Not used in the example shown# os.chdir(C:甥敳獲HilakDesktopINTERESTSMachine Learning A-Z Template FolderPart 8 - Deep LearningSection 39 - Artificial Neural Networks (ANN));# dataset pd.read_csv(Churn_Modelling.csv)# X dataset.iloc[:, 3:13].values# y dataset.iloc[:, 13].values# # Encoding categorical data# from sklearn.preprocessing import LabelEncoder, OneHotEncoder# labelencoder_X_1 LabelEncoder()# X[:, 1] labelencoder_X_1.fit_transform(X[:, 1])# labelencoder_X_2 LabelEncoder()# X[:, 2] labelencoder_X_2.fit_transform(X[:, 2])# onehotencoder OneHotEncoder(categorical_features [1])# X onehotencoder.fit_transform(X).toarray()# X X[:, 1:]# # Splitting the dataset into the Training set and Test set# from sklearn.model_selection import train_test_split# X_train, X_test, y_train, y_test train_test_split(X, y, test_size 0.2)# X_test, X_CV, y_test, y_CV train_test_split(X, y, test_size 0.5)# # Feature Scaling# from sklearn.preprocessing import StandardScaler# sc StandardScaler()# X_train sc.fit_transform(X_train)# X_test sc.transform(X_test)# X_train X_train.T# X_test X_test.T# X_CV X_CV.Tdef sigmoid(z) : Reutrns the element wise sigmoid function. return 1./(1 np.exp(-z))def sigmoid_prime(z) : Returns the derivative of the sigmoid function. return sigmoid(z)*(1-sigmoid(z))def ReLU(z) : Reutrns the element wise ReLU function. return (z*(z 0))def ReLU_prime(z) : Returns the derivative of the ReLU function. return 1*(z0)def lReLU(z) : Reutrns the element wise leaky ReLU function. return np.maximum(z/100,z)def lReLU_prime(z) : Returns the derivative of the leaky ReLU function. z 1*(z0) z[z0] 1/100 return zdef tanh(z) : Reutrns the element wise hyperbolic tangent function. return np.tanh(z)def tanh_prime(z) : Returns the derivative of the tanh function. return (1-tanh(z)**2)# A dictionary of our activation functionsPHI {sigmoid:sigmoid, relu:ReLU, lrelu:lReLU, tanh:tanh}# A dictionary containing the derivatives of our activation functionsPHI_PRIME {sigmoid:sigmoid_prime, relu:ReLU_prime, lrelu:lReLU_prime, tanh:tanh_prime}class NeuralNet : This is a class for making Artificial Neural Networks. L2 and Droupout are the default regularization methods implemented in this class. It takes the following parameters: 1. layers : A python list containing the different number of neurons in each layer. (containing the output layer) Eg - [64,32,16,16,1] 2. X : Matrix of features with rows as features and columns as different examples. 3. y : Numpy array containing the ouputs of coresponding examples. 4. ac_funcs : A python list containing activation function of each layer. Eg - [relu,relu,lrelu,tanh,sigmoid] 5. init_method : Meathod to initialize weights of the network. Can be gaussian,random,zeros. 6. loss_func : Currently not implemented 7. W : Weights of a pretrained neural network with same architecture. 8. W : Biases of a pretrained neural network with same architecture. def __init__(self, layers, X, y, ac_funcs, init_methodgaussian, loss_funcb_ce, Wnp.array([]), Bnp.array([])) : Initialize the network. # Store the layers of the network self.layers layers # ---- self.W None self.B None # Store the number of examples in the dataset as m self.m X.shape[1] # Store the full layer list as n self.n [X.shape[0], *layers] # Save the dataset self.X X # Save coresponding output self.y y # List to store the cost of the model calculated during training self.cost [] # Stores the accuracy obtained on the test set. self.acc 0 # Activation function of each layer self.ac_funcs ac_funcs self.loss loss_func # Inittialize the weights by provided methods if not provided. if len(W) and len(B) : self.W W self.B B else : if init_methodgaussian: self.W [np.random.randn(self.n[nl], self.n[nl-1]) for nl in range(1,len(self.n))] self.B [np.zeros((nl,1), float32) for nl in self.layers] elif init_method random: self.W [np.random.rand(self.n[nl], self.n[nl-1]) for nl in range(1,len(self.n))] self.B [np.random.rand(nl,1) for nl in self.layers] elif init_method zeros: self.W [np.zeros((self.n[nl], self.n[nl-1]), float32) for nl in range(1,len(self.n))] self.B [np.zeros((nl,1), float32) for nl in self.layers] def startTraining(self, epochs, alpha, _lambda, keep_prob0.5, interval100): Start training the neural network. It takes the followng parameters : 1. epochs : Number of epochs for which you want to train the network. 2. alpha : The learning rate of your network. 3. _lambda : L2 regularization parameter or the penalization parameter. 4. keep_prob : Dropout regularization parameter. The probability of neurons to deactivate. Eg - 0.8 means 20% of the neurons have been deactivated. 5. interval : The interval between updates of cost and accuracy. start time.time() for i in range(epochs1) : z,a self._feedForward(keep_prob) delta self._cost_derivative(a[-1]) for l in range(1,len(z)) : delta_w np.dot(delta, a[-l-1].T) (_lambda)*self.W[-l] delta_b np.sum(delta, axis1, keepdimsTrue) delta np.dot(self.W[-l].T, delta)*PHI_PRIME[self.ac_funcs[-l-1]](z[-l-1]) self.W[-l] self.W[-l] - (alpha/self.m)*delta_w self.B[-l] self.B[-l] - (alpha/self.m)*delta_b delta_w np.dot(delta, self.X.T ) (_lambda)*self.W[0] delta_b np.sum(delta, axis1, keepdimsTrue) self.W[0] self.W[0] - (alpha/self.m)*delta_w self.B[0] self.B[0] - (alpha/self.m)*delta_b if not i%interval : aa self.predict(self.X) if self.loss b_ce: aa aa 0.5 self.acc sum(sum(aa self.y)) / self.m cost_val self._cost_func(a[-1], _lambda) self.cost.append(cost_val) elif self.loss c_ce: aa np.argmax(aa, axis 0) yy np.argmax(self.y, axis 0) self.acc np.sum(aayy)/(self.m) cost_val self._cost_func(a[-1], _lambda) self.cost.append(cost_val) sys.stdout.write(fEpoch[{i}] : Cost {cost_val:.2f} ; Acc {(self.acc*100):.2f}% ; Time Taken {(time.time()-start):.2f}s) print() return None def predict(self, X_test) : Predict the labels for a new dataset. Returns probability. a PHI[self.ac_funcs[0]](np.dot(self.W[0], X_test) self.B[0]) for l in range(1,len(self.layers)): a PHI[self.ac_funcs[l]](np.dot(self.W[l], a) self.B[l]) return a def _feedForward(self, keep_prob): Forward pass z [];a [] z.append(np.dot(self.W[0], self.X) self.B[0]) a.append(PHI[self.ac_funcs[0]](z[-1])) for l in range(1,len(self.layers)): z.append(np.dot(self.W[l], a[-1]) self.B[l]) # a.append(PHI[self.ac_funcs[l]](z[l])) _a PHI[self.ac_funcs[l]](z[l]) a.append( ((np.random.rand(_a.shape[0],1) keep_prob)*_a)/keep_prob ) return z,a def _cost_func(self, a, _lambda): Binary Cross Entropy Cost Function return ( (-1/self.m)*np.sum(np.nan_to_num(self.y*np.log(a) (1-self.y)*np.log(1-a))) (_lambda/(2*self.m))*np.sum([np.sum(i**2) for i in self.W]) ) def _cost_derivative(self, a) : The derivative of cost w.r.t z return (a-self.y) property def summary(self) : return self.cost, self.acc, self.W,self.B def __repr__(self) : return f# Initializing our neural networkneural_net_sigmoid NeuralNet([32,16,1], X_train, y_train, ac_funcs [relu,relu,sigmoid])# Staring the training of our network.neural_net_sigmoid.startTraining(5000, 0.01, 0.2, 0.5, 100)# Predicting on new dataset using our trained network.preds neural_net_sigmoid.predict(X_test)preds preds 0.5acc (sum(sum(preds y_test)) / y_test.size)*100# Accuracy (metric of evaluation) obtained by the network.print(fTest set Accuracy ( r-t-s ) : {acc}%)# Plotting our cost vs epochs relationshipsigmoid_summary neural_net_sigmoid.summaryplt.plot(range(len(sigmoid_summary[0])), sigmoid_summary[0], labelSigmoid Cost)plt.title(Cost)plt.show()# Comparing our results with the library keras.from keras.models import Sequentialfrom keras.layers import DenseX_train, X_test X_train.T, X_test.Tclassifier Sequential()classifier.add(Dense(input_dim4, units 32, kernel_initializeruniform
