Notebook 13: Using Deep Learning to Study SUSY with Pytorch

Learning Goals

The goal of this notebook is to introduce the powerful PyTorch framework for building neural networks and use it to analyze the SUSY dataset. After this notebook, the reader should understand the mechanics of PyTorch and how to construct DNNs using this package. In addition, the reader is encouraged to explore the GPU backend available in Pytorch on this dataset.

Overview

In this notebook, we use Deep Neural Networks to classify the supersymmetry dataset, first introduced by Baldi et al. in Nature Communication (2015). The SUSY data set consists of 5,000,000 Monte-Carlo samples of supersymmetric and non-supersymmetric collisions with $18$ features. The signal process is the production of electrically-charged supersymmetric particles which decay to $W$ bosons and an electrically-neutral supersymmetric particle that is invisible to the detector.

The first $8$ features are "raw" kinematic features that can be directly measured from collisions. The final $10$ features are "hand constructed" features that have been chosen using physical knowledge and are known to be important in distinguishing supersymmetric and non-supersymmetric collision events. More specifically, they are given by the column names below.

In this notebook, we study this dataset using Pytorch.

from __future__ import print_function, division
import os,sys
import numpy as np
import torch # pytorch package, allows using GPUs
# fix seed
seed=17
np.random.seed(seed)
torch.manual_seed(seed)

<torch._C.Generator at 0x108362670>

Structure of the Procedure

Constructing a Deep Neural Network to solve ML problems is a multiple-stage process. Quite generally, one can identify the key steps as follows:

• step 1: Load and process the data
• step 2: Define the model and its architecture
• step 3: Choose the optimizer and the cost function
• step 4: Train the model
• step 5: Evaluate the model performance on the unseen test data
• step 6: Modify the hyperparameters to optimize performance for the specific data set

Below, we sometimes combine some of these steps together for convenience.

Notice that we take a rather different approach, compared to the simpler MNIST Keras notebook. We first define a set of classes and functions and run the actual computation only in the very end.

Step 1: Load and Process the SUSY Dataset

The supersymmetry dataset can be downloaded from the UCI Machine Learning repository on https://archive.ics.uci.edu/ml/machine-learning-databases/00279/SUSY.csv.gz. The dataset is quite large. Download the dataset and unzip it in a directory.

Loading data in Pytroch is done by creating a user-defined a class, which we name SUSY_Dataset, and is a child of the torch.utils.data.Dataset class. This ensures that all necessary attributes required for the processing of the data during the training and test stages are easily inherited. The __init__ method of our custom data class should contain the usual code for loading the data, which is problem-specific, and has been discussed for the SUSY data set in Notebook 5. More importantly, the user-defined data class must override the __len__ and __getitem__ methods of the parent DataSet class. The former returns the size of the data set, while the latter allows the user to access a particular data point from the set by specifying its index.

from torchvision import datasets # load data

class SUSY_Dataset(torch.utils.data.Dataset):
    """SUSY pytorch dataset."""

    def __init__(self, data_file, root_dir, dataset_size, train=True, transform=None, high_level_feats=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            train (bool, optional): If set to `True` load training data.
            transform (callable, optional): Optional transform to be applied on a sample.
            high_level_festures (bool, optional): If set to `True`, working with high-level features only. 
                                        If set to `False`, working with low-level features only.
                                        Default is `None`: working with all features
        """

        import pandas as pd

        features=['SUSY','lepton 1 pT', 'lepton 1 eta', 'lepton 1 phi', 'lepton 2 pT', 'lepton 2 eta', 'lepton 2 phi', 
                'missing energy magnitude', 'missing energy phi', 'MET_rel', 'axial MET', 'M_R', 'M_TR_2', 'R', 'MT2', 
                'S_R', 'M_Delta_R', 'dPhi_r_b', 'cos(theta_r1)']

        low_features=['lepton 1 pT', 'lepton 1 eta', 'lepton 1 phi', 'lepton 2 pT', 'lepton 2 eta', 'lepton 2 phi', 
                'missing energy magnitude', 'missing energy phi']

        high_features=['MET_rel', 'axial MET', 'M_R', 'M_TR_2', 'R', 'MT2','S_R', 'M_Delta_R', 'dPhi_r_b', 'cos(theta_r1)']


        #Number of datapoints to work with
        df = pd.read_csv(root_dir+data_file, header=None,nrows=dataset_size,engine='python')
        df.columns=features
        Y = df['SUSY']
        X = df[[col for col in df.columns if col!="SUSY"]]

        # set training and test data size
        train_size=int(0.8*dataset_size)
        self.train=train

        if self.train:
            X=X[:train_size]
            Y=Y[:train_size]
            print("Training on {} examples".format(train_size))
        else:
            X=X[train_size:]
            Y=Y[train_size:]
            print("Testing on {} examples".format(dataset_size-train_size))


        self.root_dir = root_dir
        self.transform = transform

        # make datasets using only the 8 low-level features and 10 high-level features
        if high_level_feats is None:
            self.data=(X.values.astype(np.float32),Y.values.astype(int))
            print("Using both high and low level features")
        elif high_level_feats is True:
            self.data=(X[high_features].values.astype(np.float32),Y.values.astype(int))
            print("Using both high-level features only.")
        elif high_level_feats is False:
            self.data=(X[low_features].values.astype(np.float32),Y.values.astype(int))
            print("Using both low-level features only.")


    # override __len__ and __getitem__ of the Dataset() class

    def __len__(self):
        return len(self.data[1])

    def __getitem__(self, idx):

        sample=(self.data[0][idx,...],self.data[1][idx])

        if self.transform:
            sample=self.transform(sample)

        return sample

Last, we define a helper function load_data() that accepts as a required argument the set of parameters args, and returns two generators: test_loader and train_loader which readily return mini-batches.

def load_data(args):

    data_file='SUSY.csv'
    root_dir=os.path.expanduser('~')+'/ML_review/SUSY_data/'

    kwargs = {} # CUDA arguments, if enabled
    # load and noralise train and test data
    train_loader = torch.utils.data.DataLoader(
        SUSY_Dataset(data_file,root_dir,args.dataset_size,train=True,high_level_feats=args.high_level_feats),
        batch_size=args.batch_size, shuffle=True, **kwargs)

    test_loader = torch.utils.data.DataLoader(
        SUSY_Dataset(data_file,root_dir,args.dataset_size,train=False,high_level_feats=args.high_level_feats),
        batch_size=args.test_batch_size, shuffle=True, **kwargs)

    return train_loader, test_loader

Step 2: Define the Neural Net and its Architecture

To construct neural networks with Pytorch, we make another class called model as a child of Pytorch's nn.Module class. The model class initializes the types of layers needed for the deep neural net in its __init__ method, while the DNN is assembled in a function method called forward, which accepts an autograd.Variable object and returns the output layer. Using this convention Pytorch will automatically recognize the structure of the DNN, and the autograd module will pull the gradients forward and backward using backprop.

Our code below is constructed in such a way that one can choose whether to use the high-level and low-level features separately and altogether. This choice determines the size of the fully-connected input layer fc1. Therefore the __init__ method accepts the optional argument high_level_feats.

import torch.nn as nn # construct NN

class model(nn.Module):
    def __init__(self,high_level_feats=None):
        # inherit attributes and methods of nn.Module
        super(model, self).__init__()

        # an affine operation: y = Wx + b
        if high_level_feats is None:
            self.fc1 = nn.Linear(18, 200) # all features
        elif high_level_feats:
            self.fc1 = nn.Linear(10, 200) # low-level only
        else:
            self.fc1 = nn.Linear(8, 200) # high-level only


        self.batchnorm1=nn.BatchNorm1d(200, eps=1e-05, momentum=0.1)
        self.batchnorm2=nn.BatchNorm1d(100, eps=1e-05, momentum=0.1)

        self.fc2 = nn.Linear(200, 100) # see forward function for dimensions
        self.fc3 = nn.Linear(100, 2)

    def forward(self, x):
        '''Defines the feed-forward function for the NN.

        A backward function is automatically defined using `torch.autograd`

        Parameters
        ----------
        x : autograd.Tensor
            input data

        Returns
        -------
        autograd.Tensor
            output layer of NN

        '''

        # apply rectified linear unit
        x = F.relu(self.fc1(x))
        # apply dropout
        #x=self.batchnorm1(x)
        x = F.dropout(x, training=self.training)


        # apply rectified linear unit
        x = F.relu(self.fc2(x))
        # apply dropout
        #x=self.batchnorm2(x)
        x = F.dropout(x, training=self.training)


        # apply affine operation fc2
        x = self.fc3(x)
        # soft-max layer
        x = F.log_softmax(x,dim=1)

        return x

Steps 3+4+5: Choose the Optimizer and the Cost Function. Train and Evaluate the Model

Next, we define the function evaluate_model. The first argument, args, contains all hyperparameters needed for the DNN (see below). The second and third arguments are the train_loader and the test_loader objects, returned by the function load_data() we defined in Step 1 above. The evaluate_model function returns the final test_loss and test_accuracy of the model.

First, we initialize a model and call the object DNN. In order to define the loss function and the optimizer, we use modules torch.nn.functional (imported here as F) and torch.optim. As a loss function we choose the negative log-likelihood, and stored is under the variable criterion. As usual, we can choose any from a variety of different SGD-based optimizers, but we focus on the traditional SGD.

Next, we define two functions: train() and test(). They are called at the end of evaluate_model where we loop over the training epochs to train and test our model.

The train function accepts an integer called epoch, which is only used to print the training data. We first set the DNN in a train mode using the train() method inherited from nn.Module. Then we loop over the mini-batches in train_loader. We cast the data as pytorch Variable, re-set the optimizer, perform the forward step by calling the DNN model on the data and computing the loss. The backprop algorithm is then easily done using the backward() method of the loss function criterion. We use optimizer.step to update the weights of the DNN. Last print the performance for every minibatch. train returns the loss on the data.

The test function is similar to train but its purpose is to test the performance of a trained model. Once we set the DNN model in eval() mode, the following steps are similar to those in train. We then compute the test_loss and the number of correct predictions, print the results and return them.

import torch.nn.functional as F # implements forward and backward definitions of an autograd operation
import torch.optim as optim # different update rules such as SGD, Nesterov-SGD, Adam, RMSProp, etc

def evaluate_model(args,train_loader,test_loader):

    # create model
    DNN = model(high_level_feats=args.high_level_feats)
    # negative log-likelihood (nll) loss for training: takes class labels NOT one-hot vectors!
    criterion = F.nll_loss
    # define SGD optimizer
    optimizer = optim.SGD(DNN.parameters(), lr=args.lr, momentum=args.momentum)
    #optimizer = optim.Adam(DNN.parameters(), lr=0.001, betas=(0.9, 0.999))


    ################################################

    def train(epoch):
        '''Trains a NN using minibatches.

        Parameters
        ----------
        epoch : int
            Training epoch number.

        '''

        # set model to training mode (affects Dropout and BatchNorm)
        DNN.train()
        # loop over training data
        for batch_idx, (data, label) in enumerate(train_loader):
            # zero gradient buffers
            optimizer.zero_grad()
            # compute output of final layer: forward step
            output = DNN(data)
            # compute loss
            loss = criterion(output, label)
            # run backprop: backward step
            loss.backward()
            # update weigths of NN
            optimizer.step()
            
            # print loss at current epoch
            if batch_idx % args.log_interval == 0:
                print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                    epoch, batch_idx * len(data), len(train_loader.dataset),
                    100. * batch_idx / len(train_loader), loss.item() ))
            

        return loss.item()

    ################################################

    def test():
        '''Tests NN performance.

        '''

        # evaluate model
        DNN.eval()

        test_loss = 0 # loss function on test data
        correct = 0 # number of correct predictions
        # loop over test data
        for data, label in test_loader:
            # compute model prediction softmax probability
            output = DNN(data)
            # compute test loss
            test_loss += criterion(output, label, size_average=False).item() # sum up batch loss
            # find most likely prediction
            pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
            # update number of correct predictions
            correct += pred.eq(label.data.view_as(pred)).cpu().sum().item()

        # print test loss
        test_loss /= len(test_loader.dataset)
        
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.3f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))
        

        return test_loss, correct / len(test_loader.dataset)


    ################################################


    train_loss=np.zeros((args.epochs,))
    test_loss=np.zeros_like(train_loss)
    test_accuracy=np.zeros_like(train_loss)

    epochs=range(1, args.epochs + 1)
    for epoch in epochs:

        train_loss[epoch-1] = train(epoch)
        test_loss[epoch-1], test_accuracy[epoch-1] = test()



    return test_loss[-1], test_accuracy[-1]

Step 6: Modify the Hyperparameters to Optimize Performance of the Model

To study the performance of the model for a variety of different data_set_sizes and learning_rates, we do a grid search.

Let us define a function grid_search, which accepts the args variable containing all hyper-parameters needed for the problem. After choosing logarithmically-spaced data_set_sizes and learning_rates, we first loop over all data_set_sizes, update the args variable, and call the load_data function. We then loop once again over all learning_rates, update args and call evaluate_model.

def grid_search(args):


    # perform grid search over learnign rate and number of hidden neurons
    dataset_sizes=[1000, 10000, 100000, 200000] #np.logspace(2,5,4).astype('int')
    learning_rates=np.logspace(-5,-1,5)

    # pre-alocate data
    test_loss=np.zeros((len(dataset_sizes),len(learning_rates)),dtype=np.float64)
    test_accuracy=np.zeros_like(test_loss)

    # do grid search
    for i, dataset_size in enumerate(dataset_sizes):
        # upate data set size parameters
        args.dataset_size=dataset_size
        args.batch_size=int(0.01*dataset_size)

        # load data
        train_loader, test_loader = load_data(args)

        for j, lr in enumerate(learning_rates):
            # update learning rate
            args.lr=lr

            print("\n training DNN with %5d data points and SGD lr=%0.6f. \n" %(dataset_size,lr) )

            test_loss[i,j],test_accuracy[i,j] = evaluate_model(args,train_loader,test_loader)


    plot_data(learning_rates,dataset_sizes,test_accuracy)

Last, we use the function plot_data, defined below, to plot the results.

import matplotlib.pyplot as plt

def plot_data(x,y,data):

    # plot results
    fontsize=16


    fig = plt.figure()
    ax = fig.add_subplot(111)
    cax = ax.matshow(data, interpolation='nearest', vmin=0, vmax=1)
    
    cbar=fig.colorbar(cax)
    cbar.ax.set_ylabel('accuracy (%)',rotation=90,fontsize=fontsize)
    cbar.set_ticks([0,.2,.4,0.6,0.8,1.0])
    cbar.set_ticklabels(['0%','20%','40%','60%','80%','100%'])

    # put text on matrix elements
    for i, x_val in enumerate(np.arange(len(x))):
        for j, y_val in enumerate(np.arange(len(y))):
            c = "${0:.1f}\\%$".format( 100*data[j,i])  
            ax.text(x_val, y_val, c, va='center', ha='center')

    # convert axis vaues to to string labels
    x=[str(i) for i in x]
    y=[str(i) for i in y]


    ax.set_xticklabels(['']+x)
    ax.set_yticklabels(['']+y)

    ax.set_xlabel('$\\mathrm{learning\\ rate}$',fontsize=fontsize)
    ax.set_ylabel('$\\mathrm{hidden\\ neurons}$',fontsize=fontsize)

    plt.tight_layout()

    plt.show()

Run Code

As we mentioned in the beginning of the notebook, all functions and classes discussed above only specify the procedure but do not actually perform any computations. This allows us to re-use them for different problems.

Actually running the training and testing for every point in the grid search is done below. The argparse class allows us to conveniently keep track of all hyperparameters, stored in the variable args which enters most of the functions we defined above.

To run the simulation, we call the function grid_search.

Exercises

• One of the advantages of Pytorch is that it allows to automatically use the CUDA library for fast performance on GPU's. For the sake of clarity, we have omitted this in the above notebook. Go online to check how to put the CUDA commands back into the code above. Hint: study the Pytorch MNIST tutorial to see how this works in practice.

import argparse # handles arguments
import sys; sys.argv=['']; del sys # required to use parser in jupyter notebooks

# Training settings
parser = argparse.ArgumentParser(description='PyTorch SUSY Example')
parser.add_argument('--dataset_size', type=int, default=100000, metavar='DS',
                help='size of data set (default: 100000)')
parser.add_argument('--high_level_feats', type=bool, default=None, metavar='HLF',
                help='toggles high level features (default: None)')
parser.add_argument('--batch-size', type=int, default=100, metavar='N',
                help='input batch size for training (default: 64)')
parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                help='input batch size for testing (default: 1000)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
                help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.05, metavar='LR',
                help='learning rate (default: 0.02)')
parser.add_argument('--momentum', type=float, default=0.8, metavar='M',
                help='SGD momentum (default: 0.5)')
parser.add_argument('--no-cuda', action='store_true', default=False,
                help='disables CUDA training')
parser.add_argument('--seed', type=int, default=2, metavar='S',
                help='random seed (default: 1)')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                help='how many batches to wait before logging training status')
args = parser.parse_args()

# set seed of random number generator
torch.manual_seed(args.seed)

grid_search(args)

Training on 800 examples
Using both high and low level features
Testing on 200 examples
Using both high and low level features

 training DNN with  1000 data points and SGD lr=0.000010. 

Train Epoch: 1 [0/800 (0%)]	Loss: 0.561476
Train Epoch: 1 [100/800 (12%)]	Loss: 0.823435
Train Epoch: 1 [200/800 (25%)]	Loss: 0.647225
Train Epoch: 1 [300/800 (38%)]	Loss: 0.612186
Train Epoch: 1 [400/800 (50%)]	Loss: 0.962393
Train Epoch: 1 [500/800 (62%)]	Loss: 0.835941
Train Epoch: 1 [600/800 (75%)]	Loss: 0.808794
Train Epoch: 1 [700/800 (88%)]	Loss: 0.766973

Test set: Average loss: 0.7115, Accuracy: 109/200 (54.500%)

Train Epoch: 2 [0/800 (0%)]	Loss: 0.861468
Train Epoch: 2 [100/800 (12%)]	Loss: 0.653841
Train Epoch: 2 [200/800 (25%)]	Loss: 0.823339
Train Epoch: 2 [300/800 (38%)]	Loss: 0.745887
Train Epoch: 2 [400/800 (50%)]	Loss: 0.694589
Train Epoch: 2 [500/800 (62%)]	Loss: 0.693052
Train Epoch: 2 [600/800 (75%)]	Loss: 0.719047
Train Epoch: 2 [700/800 (88%)]	Loss: 0.591686

Test set: Average loss: 0.7107, Accuracy: 109/200 (54.500%)

Train Epoch: 3 [0/800 (0%)]	Loss: 0.728128
Train Epoch: 3 [100/800 (12%)]	Loss: 0.698269
Train Epoch: 3 [200/800 (25%)]	Loss: 0.705191
Train Epoch: 3 [300/800 (38%)]	Loss: 0.683300
Train Epoch: 3 [400/800 (50%)]	Loss: 0.732665
Train Epoch: 3 [500/800 (62%)]	Loss: 0.849138
Train Epoch: 3 [600/800 (75%)]	Loss: 0.728277

/Users/mgbukov/miniconda3/envs/mlreview/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))

Train Epoch: 3 [700/800 (88%)]	Loss: 0.743678

Test set: Average loss: 0.7098, Accuracy: 109/200 (54.500%)

Train Epoch: 4 [0/800 (0%)]	Loss: 0.687534
Train Epoch: 4 [100/800 (12%)]	Loss: 0.729848
Train Epoch: 4 [200/800 (25%)]	Loss: 0.774060
Train Epoch: 4 [300/800 (38%)]	Loss: 0.767740
Train Epoch: 4 [400/800 (50%)]	Loss: 0.747602
Train Epoch: 4 [500/800 (62%)]	Loss: 0.678511
Train Epoch: 4 [600/800 (75%)]	Loss: 0.892563
Train Epoch: 4 [700/800 (88%)]	Loss: 0.617390

Test set: Average loss: 0.7091, Accuracy: 109/200 (54.500%)

Train Epoch: 5 [0/800 (0%)]	Loss: 0.666867
Train Epoch: 5 [100/800 (12%)]	Loss: 0.747424
Train Epoch: 5 [200/800 (25%)]	Loss: 0.623322
Train Epoch: 5 [300/800 (38%)]	Loss: 0.803995
Train Epoch: 5 [400/800 (50%)]	Loss: 0.729541
Train Epoch: 5 [500/800 (62%)]	Loss: 0.844938
Train Epoch: 5 [600/800 (75%)]	Loss: 0.717547
Train Epoch: 5 [700/800 (88%)]	Loss: 0.595089

Test set: Average loss: 0.7084, Accuracy: 109/200 (54.500%)

Train Epoch: 6 [0/800 (0%)]	Loss: 0.772091
Train Epoch: 6 [100/800 (12%)]	Loss: 0.641700
Train Epoch: 6 [200/800 (25%)]	Loss: 0.948149
Train Epoch: 6 [300/800 (38%)]	Loss: 0.783350
Train Epoch: 6 [400/800 (50%)]	Loss: 0.803563
Train Epoch: 6 [500/800 (62%)]	Loss: 0.749601
Train Epoch: 6 [600/800 (75%)]	Loss: 0.590462
Train Epoch: 6 [700/800 (88%)]	Loss: 0.793399

Test set: Average loss: 0.7077, Accuracy: 109/200 (54.500%)

Train Epoch: 7 [0/800 (0%)]	Loss: 0.699198
Train Epoch: 7 [100/800 (12%)]	Loss: 0.538285
Train Epoch: 7 [200/800 (25%)]	Loss: 0.637657
Train Epoch: 7 [300/800 (38%)]	Loss: 0.707620
Train Epoch: 7 [400/800 (50%)]	Loss: 0.739883
Train Epoch: 7 [500/800 (62%)]	Loss: 0.710847
Train Epoch: 7 [600/800 (75%)]	Loss: 0.883698
Train Epoch: 7 [700/800 (88%)]	Loss: 0.878761

Test set: Average loss: 0.7069, Accuracy: 109/200 (54.500%)

Train Epoch: 8 [0/800 (0%)]	Loss: 0.640105
Train Epoch: 8 [100/800 (12%)]	Loss: 0.690040
Train Epoch: 8 [200/800 (25%)]	Loss: 0.777747
Train Epoch: 8 [300/800 (38%)]	Loss: 0.596149
Train Epoch: 8 [400/800 (50%)]	Loss: 0.601473
Train Epoch: 8 [500/800 (62%)]	Loss: 0.621435
Train Epoch: 8 [600/800 (75%)]	Loss: 0.908196
Train Epoch: 8 [700/800 (88%)]	Loss: 0.859109

Test set: Average loss: 0.7062, Accuracy: 109/200 (54.500%)

Train Epoch: 9 [0/800 (0%)]	Loss: 0.566881
Train Epoch: 9 [100/800 (12%)]	Loss: 0.710102
Train Epoch: 9 [200/800 (25%)]	Loss: 0.788556
Train Epoch: 9 [300/800 (38%)]	Loss: 0.702322
Train Epoch: 9 [400/800 (50%)]	Loss: 0.720015
Train Epoch: 9 [500/800 (62%)]	Loss: 0.645902
Train Epoch: 9 [600/800 (75%)]	Loss: 0.741547
Train Epoch: 9 [700/800 (88%)]	Loss: 0.765223

Test set: Average loss: 0.7055, Accuracy: 109/200 (54.500%)

Train Epoch: 10 [0/800 (0%)]	Loss: 0.819819
Train Epoch: 10 [100/800 (12%)]	Loss: 0.629852
Train Epoch: 10 [200/800 (25%)]	Loss: 0.679486
Train Epoch: 10 [300/800 (38%)]	Loss: 0.681212
Train Epoch: 10 [400/800 (50%)]	Loss: 0.841154
Train Epoch: 10 [500/800 (62%)]	Loss: 0.610462
Train Epoch: 10 [600/800 (75%)]	Loss: 0.742932
Train Epoch: 10 [700/800 (88%)]	Loss: 0.743243

Test set: Average loss: 0.7048, Accuracy: 109/200 (54.500%)


 training DNN with  1000 data points and SGD lr=0.000100. 

Train Epoch: 1 [0/800 (0%)]	Loss: 0.707171
Train Epoch: 1 [100/800 (12%)]	Loss: 0.759615
Train Epoch: 1 [200/800 (25%)]	Loss: 0.623426
Train Epoch: 1 [300/800 (38%)]	Loss: 0.761207
Train Epoch: 1 [400/800 (50%)]	Loss: 0.690813
Train Epoch: 1 [500/800 (62%)]	Loss: 0.776441
Train Epoch: 1 [600/800 (75%)]	Loss: 0.685847
Train Epoch: 1 [700/800 (88%)]	Loss: 0.706618

Test set: Average loss: 0.6949, Accuracy: 97/200 (48.500%)

Train Epoch: 2 [0/800 (0%)]	Loss: 0.734724
Train Epoch: 2 [100/800 (12%)]	Loss: 0.769788
Train Epoch: 2 [200/800 (25%)]	Loss: 0.714627
Train Epoch: 2 [300/800 (38%)]	Loss: 0.700415
Train Epoch: 2 [400/800 (50%)]	Loss: 0.721317
Train Epoch: 2 [500/800 (62%)]	Loss: 0.725358
Train Epoch: 2 [600/800 (75%)]	Loss: 0.728227
Train Epoch: 2 [700/800 (88%)]	Loss: 0.659629

Test set: Average loss: 0.6933, Accuracy: 96/200 (48.000%)

Train Epoch: 3 [0/800 (0%)]	Loss: 0.795568
Train Epoch: 3 [100/800 (12%)]	Loss: 0.696280
Train Epoch: 3 [200/800 (25%)]	Loss: 0.744723
Train Epoch: 3 [300/800 (38%)]	Loss: 0.661478
Train Epoch: 3 [400/800 (50%)]	Loss: 0.653667
Train Epoch: 3 [500/800 (62%)]	Loss: 0.740828
Train Epoch: 3 [600/800 (75%)]	Loss: 0.715702
Train Epoch: 3 [700/800 (88%)]	Loss: 0.647341

Test set: Average loss: 0.6917, Accuracy: 103/200 (51.500%)

Train Epoch: 4 [0/800 (0%)]	Loss: 0.718259
Train Epoch: 4 [100/800 (12%)]	Loss: 0.715631
Train Epoch: 4 [200/800 (25%)]	Loss: 0.678426
Train Epoch: 4 [300/800 (38%)]	Loss: 0.707464
Train Epoch: 4 [400/800 (50%)]	Loss: 0.709158
Train Epoch: 4 [500/800 (62%)]	Loss: 0.740914
Train Epoch: 4 [600/800 (75%)]	Loss: 0.697477
Train Epoch: 4 [700/800 (88%)]	Loss: 0.785577

Test set: Average loss: 0.6903, Accuracy: 99/200 (49.500%)

Train Epoch: 5 [0/800 (0%)]	Loss: 0.712337
Train Epoch: 5 [100/800 (12%)]	Loss: 0.658852
Train Epoch: 5 [200/800 (25%)]	Loss: 0.690872
Train Epoch: 5 [300/800 (38%)]	Loss: 0.664473
Train Epoch: 5 [400/800 (50%)]	Loss: 0.672096
Train Epoch: 5 [500/800 (62%)]	Loss: 0.681344
Train Epoch: 5 [600/800 (75%)]	Loss: 0.745848
Train Epoch: 5 [700/800 (88%)]	Loss: 0.712893

Test set: Average loss: 0.6890, Accuracy: 94/200 (47.000%)

Train Epoch: 6 [0/800 (0%)]	Loss: 0.720909
Train Epoch: 6 [100/800 (12%)]	Loss: 0.730349
Train Epoch: 6 [200/800 (25%)]	Loss: 0.763710
Train Epoch: 6 [300/800 (38%)]	Loss: 0.727711
Train Epoch: 6 [400/800 (50%)]	Loss: 0.672487
Train Epoch: 6 [500/800 (62%)]	Loss: 0.677815
Train Epoch: 6 [600/800 (75%)]	Loss: 0.626029
Train Epoch: 6 [700/800 (88%)]	Loss: 0.706892

Test set: Average loss: 0.6877, Accuracy: 95/200 (47.500%)

Train Epoch: 7 [0/800 (0%)]	Loss: 0.762181
Train Epoch: 7 [100/800 (12%)]	Loss: 0.655490
Train Epoch: 7 [200/800 (25%)]	Loss: 0.785486
Train Epoch: 7 [300/800 (38%)]	Loss: 0.719932
Train Epoch: 7 [400/800 (50%)]	Loss: 0.683267
Train Epoch: 7 [500/800 (62%)]	Loss: 0.716905
Train Epoch: 7 [600/800 (75%)]	Loss: 0.610312
Train Epoch: 7 [700/800 (88%)]	Loss: 0.689631

Test set: Average loss: 0.6864, Accuracy: 97/200 (48.500%)

Train Epoch: 8 [0/800 (0%)]	Loss: 0.718268
Train Epoch: 8 [100/800 (12%)]	Loss: 0.626109
Train Epoch: 8 [200/800 (25%)]	Loss: 0.694751
Train Epoch: 8 [300/800 (38%)]	Loss: 0.777661
Train Epoch: 8 [400/800 (50%)]	Loss: 0.642748
Train Epoch: 8 [500/800 (62%)]	Loss: 0.734109
Train Epoch: 8 [600/800 (75%)]	Loss: 0.733591
Train Epoch: 8 [700/800 (88%)]	Loss: 0.704688

Test set: Average loss: 0.6851, Accuracy: 99/200 (49.500%)

Train Epoch: 9 [0/800 (0%)]	Loss: 0.645388
Train Epoch: 9 [100/800 (12%)]	Loss: 0.730897
Train Epoch: 9 [200/800 (25%)]	Loss: 0.745258
Train Epoch: 9 [300/800 (38%)]	Loss: 0.746987
Train Epoch: 9 [400/800 (50%)]	Loss: 0.675306
Train Epoch: 9 [500/800 (62%)]	Loss: 0.759842
Train Epoch: 9 [600/800 (75%)]	Loss: 0.723907
Train Epoch: 9 [700/800 (88%)]	Loss: 0.661631

Test set: Average loss: 0.6839, Accuracy: 102/200 (51.000%)

Train Epoch: 10 [0/800 (0%)]	Loss: 0.637127
Train Epoch: 10 [100/800 (12%)]	Loss: 0.703343
Train Epoch: 10 [200/800 (25%)]	Loss: 0.625676
Train Epoch: 10 [300/800 (38%)]	Loss: 0.654124
Train Epoch: 10 [400/800 (50%)]	Loss: 0.675645
Train Epoch: 10 [500/800 (62%)]	Loss: 0.675741
Train Epoch: 10 [600/800 (75%)]	Loss: 0.694248
Train Epoch: 10 [700/800 (88%)]	Loss: 0.652979

Test set: Average loss: 0.6826, Accuracy: 103/200 (51.500%)


 training DNN with  1000 data points and SGD lr=0.001000. 

Train Epoch: 1 [0/800 (0%)]	Loss: 0.828281
Train Epoch: 1 [100/800 (12%)]	Loss: 0.706446
Train Epoch: 1 [200/800 (25%)]	Loss: 0.742615
Train Epoch: 1 [300/800 (38%)]	Loss: 0.699289
Train Epoch: 1 [400/800 (50%)]	Loss: 0.695920
Train Epoch: 1 [500/800 (62%)]	Loss: 0.700937
Train Epoch: 1 [600/800 (75%)]	Loss: 0.670519
Train Epoch: 1 [700/800 (88%)]	Loss: 0.704970

Test set: Average loss: 0.6916, Accuracy: 101/200 (50.500%)

Train Epoch: 2 [0/800 (0%)]	Loss: 0.707742
Train Epoch: 2 [100/800 (12%)]	Loss: 0.679032
Train Epoch: 2 [200/800 (25%)]	Loss: 0.703815
Train Epoch: 2 [300/800 (38%)]	Loss: 0.641242
Train Epoch: 2 [400/800 (50%)]	Loss: 0.631897
Train Epoch: 2 [500/800 (62%)]	Loss: 0.673181
Train Epoch: 2 [600/800 (75%)]	Loss: 0.758181
Train Epoch: 2 [700/800 (88%)]	Loss: 0.686858

Test set: Average loss: 0.6833, Accuracy: 123/200 (61.500%)

Train Epoch: 3 [0/800 (0%)]	Loss: 0.666551
Train Epoch: 3 [100/800 (12%)]	Loss: 0.738501
Train Epoch: 3 [200/800 (25%)]	Loss: 0.736996
Train Epoch: 3 [300/800 (38%)]	Loss: 0.663481
Train Epoch: 3 [400/800 (50%)]	Loss: 0.715448
Train Epoch: 3 [500/800 (62%)]	Loss: 0.649770
Train Epoch: 3 [600/800 (75%)]	Loss: 0.677709
Train Epoch: 3 [700/800 (88%)]	Loss: 0.649908

Test set: Average loss: 0.6751, Accuracy: 133/200 (66.500%)

Train Epoch: 4 [0/800 (0%)]	Loss: 0.638706
Train Epoch: 4 [100/800 (12%)]	Loss: 0.646711
Train Epoch: 4 [200/800 (25%)]	Loss: 0.738700
Train Epoch: 4 [300/800 (38%)]	Loss: 0.629142
Train Epoch: 4 [400/800 (50%)]	Loss: 0.624597
Train Epoch: 4 [500/800 (62%)]	Loss: 0.745657
Train Epoch: 4 [600/800 (75%)]	Loss: 0.725137
Train Epoch: 4 [700/800 (88%)]	Loss: 0.640283

Test set: Average loss: 0.6675, Accuracy: 140/200 (70.000%)

Train Epoch: 5 [0/800 (0%)]	Loss: 0.687889
Train Epoch: 5 [100/800 (12%)]	Loss: 0.650391
Train Epoch: 5 [200/800 (25%)]	Loss: 0.646154
Train Epoch: 5 [300/800 (38%)]	Loss: 0.654916
Train Epoch: 5 [400/800 (50%)]	Loss: 0.679761
Train Epoch: 5 [500/800 (62%)]	Loss: 0.631989
Train Epoch: 5 [600/800 (75%)]	Loss: 0.691708
Train Epoch: 5 [700/800 (88%)]	Loss: 0.691091

Test set: Average loss: 0.6597, Accuracy: 146/200 (73.000%)

Train Epoch: 6 [0/800 (0%)]	Loss: 0.619725
Train Epoch: 6 [100/800 (12%)]	Loss: 0.687790
Train Epoch: 6 [200/800 (25%)]	Loss: 0.592156
Train Epoch: 6 [300/800 (38%)]	Loss: 0.695230
Train Epoch: 6 [400/800 (50%)]	Loss: 0.663265
Train Epoch: 6 [500/800 (62%)]	Loss: 0.637461
Train Epoch: 6 [600/800 (75%)]	Loss: 0.758096
Train Epoch: 6 [700/800 (88%)]	Loss: 0.589519

Test set: Average loss: 0.6530, Accuracy: 144/200 (72.000%)

Train Epoch: 7 [0/800 (0%)]	Loss: 0.720149
Train Epoch: 7 [100/800 (12%)]	Loss: 0.675038
Train Epoch: 7 [200/800 (25%)]	Loss: 0.624679
Train Epoch: 7 [300/800 (38%)]	Loss: 0.696514
Train Epoch: 7 [400/800 (50%)]	Loss: 0.602463
Train Epoch: 7 [500/800 (62%)]	Loss: 0.738410
Train Epoch: 7 [600/800 (75%)]	Loss: 0.722651
Train Epoch: 7 [700/800 (88%)]	Loss: 0.631086

Test set: Average loss: 0.6458, Accuracy: 145/200 (72.500%)

Train Epoch: 8 [0/800 (0%)]	Loss: 0.564008
Train Epoch: 8 [100/800 (12%)]	Loss: 0.686089
Train Epoch: 8 [200/800 (25%)]	Loss: 0.735220
Train Epoch: 8 [300/800 (38%)]	Loss: 0.681267
Train Epoch: 8 [400/800 (50%)]	Loss: 0.534413
Train Epoch: 8 [500/800 (62%)]	Loss: 0.631500
Train Epoch: 8 [600/800 (75%)]	Loss: 0.675105
Train Epoch: 8 [700/800 (88%)]	Loss: 0.622950

Test set: Average loss: 0.6396, Accuracy: 150/200 (75.000%)

Train Epoch: 9 [0/800 (0%)]	Loss: 0.716496
Train Epoch: 9 [100/800 (12%)]	Loss: 0.633340
Train Epoch: 9 [200/800 (25%)]	Loss: 0.609537
Train Epoch: 9 [300/800 (38%)]	Loss: 0.650772
Train Epoch: 9 [400/800 (50%)]	Loss: 0.756648
Train Epoch: 9 [500/800 (62%)]	Loss: 0.684104
Train Epoch: 9 [600/800 (75%)]	Loss: 0.657388
Train Epoch: 9 [700/800 (88%)]	Loss: 0.570877

Test set: Average loss: 0.6319, Accuracy: 146/200 (73.000%)

Train Epoch: 10 [0/800 (0%)]	Loss: 0.550361
Train Epoch: 10 [100/800 (12%)]	Loss: 0.661233
Train Epoch: 10 [200/800 (25%)]	Loss: 0.817553
Train Epoch: 10 [300/800 (38%)]	Loss: 0.591799
Train Epoch: 10 [400/800 (50%)]	Loss: 0.735668
Train Epoch: 10 [500/800 (62%)]	Loss: 0.642043
Train Epoch: 10 [600/800 (75%)]	Loss: 0.671387
Train Epoch: 10 [700/800 (88%)]	Loss: 0.631629

Test set: Average loss: 0.6257, Accuracy: 149/200 (74.500%)


 training DNN with  1000 data points and SGD lr=0.010000. 

Train Epoch: 1 [0/800 (0%)]	Loss: 0.696035
Train Epoch: 1 [100/800 (12%)]	Loss: 0.618539
Train Epoch: 1 [200/800 (25%)]	Loss: 0.718540
Train Epoch: 1 [300/800 (38%)]	Loss: 0.650360
Train Epoch: 1 [400/800 (50%)]	Loss: 0.642039
Train Epoch: 1 [500/800 (62%)]	Loss: 0.626914
Train Epoch: 1 [600/800 (75%)]	Loss: 0.624919
Train Epoch: 1 [700/800 (88%)]	Loss: 0.514577

Test set: Average loss: 0.5904, Accuracy: 142/200 (71.000%)

Train Epoch: 2 [0/800 (0%)]	Loss: 0.682929
Train Epoch: 2 [100/800 (12%)]	Loss: 0.651318
Train Epoch: 2 [200/800 (25%)]	Loss: 0.588398
Train Epoch: 2 [300/800 (38%)]	Loss: 0.664375
Train Epoch: 2 [400/800 (50%)]	Loss: 0.680439
Train Epoch: 2 [500/800 (62%)]	Loss: 0.693252
Train Epoch: 2 [600/800 (75%)]	Loss: 0.840514
Train Epoch: 2 [700/800 (88%)]	Loss: 0.851084

Test set: Average loss: 0.5447, Accuracy: 149/200 (74.500%)

Train Epoch: 3 [0/800 (0%)]	Loss: 0.430184
Train Epoch: 3 [100/800 (12%)]	Loss: 0.797171
Train Epoch: 3 [200/800 (25%)]	Loss: 0.413475
Train Epoch: 3 [300/800 (38%)]	Loss: 0.561503
Train Epoch: 3 [400/800 (50%)]	Loss: 0.684532
Train Epoch: 3 [500/800 (62%)]	Loss: 0.457825
Train Epoch: 3 [600/800 (75%)]	Loss: 0.595141
Train Epoch: 3 [700/800 (88%)]	Loss: 1.026458

Test set: Average loss: 0.5341, Accuracy: 149/200 (74.500%)

Train Epoch: 4 [0/800 (0%)]	Loss: 0.618329
Train Epoch: 4 [100/800 (12%)]	Loss: 0.640518
Train Epoch: 4 [200/800 (25%)]	Loss: 0.572886
Train Epoch: 4 [300/800 (38%)]	Loss: 0.420373
Train Epoch: 4 [400/800 (50%)]	Loss: 0.487304
Train Epoch: 4 [500/800 (62%)]	Loss: 0.628173
Train Epoch: 4 [600/800 (75%)]	Loss: 0.389652
Train Epoch: 4 [700/800 (88%)]	Loss: 0.448448

Test set: Average loss: 0.5121, Accuracy: 154/200 (77.000%)

Train Epoch: 5 [0/800 (0%)]	Loss: 0.809161
Train Epoch: 5 [100/800 (12%)]	Loss: 0.395011
Train Epoch: 5 [200/800 (25%)]	Loss: 0.343491
Train Epoch: 5 [300/800 (38%)]	Loss: 0.381724
Train Epoch: 5 [400/800 (50%)]	Loss: 0.409402
Train Epoch: 5 [500/800 (62%)]	Loss: 0.664123
Train Epoch: 5 [600/800 (75%)]	Loss: 0.365333
Train Epoch: 5 [700/800 (88%)]	Loss: 0.558361

Test set: Average loss: 0.5073, Accuracy: 152/200 (76.000%)

Train Epoch: 6 [0/800 (0%)]	Loss: 0.427038
Train Epoch: 6 [100/800 (12%)]	Loss: 0.497466
Train Epoch: 6 [200/800 (25%)]	Loss: 0.307992
Train Epoch: 6 [300/800 (38%)]	Loss: 0.574125
Train Epoch: 6 [400/800 (50%)]	Loss: 0.495350
Train Epoch: 6 [500/800 (62%)]	Loss: 0.407845
Train Epoch: 6 [600/800 (75%)]	Loss: 0.526377
Train Epoch: 6 [700/800 (88%)]	Loss: 0.459272

Test set: Average loss: 0.5031, Accuracy: 157/200 (78.500%)

Train Epoch: 7 [0/800 (0%)]	Loss: 0.440641
Train Epoch: 7 [100/800 (12%)]	Loss: 0.373742
Train Epoch: 7 [200/800 (25%)]	Loss: 0.546155
Train Epoch: 7 [300/800 (38%)]	Loss: 0.580418
Train Epoch: 7 [400/800 (50%)]	Loss: 0.224490
Train Epoch: 7 [500/800 (62%)]	Loss: 0.636834
Train Epoch: 7 [600/800 (75%)]	Loss: 0.505638
Train Epoch: 7 [700/800 (88%)]	Loss: 0.655799

Test set: Average loss: 0.5026, Accuracy: 150/200 (75.000%)

Train Epoch: 8 [0/800 (0%)]	Loss: 0.515595
Train Epoch: 8 [100/800 (12%)]	Loss: 0.641143
Train Epoch: 8 [200/800 (25%)]	Loss: 0.330472
Train Epoch: 8 [300/800 (38%)]	Loss: 0.406667
Train Epoch: 8 [400/800 (50%)]	Loss: 0.369609
Train Epoch: 8 [500/800 (62%)]	Loss: 0.438342
Train Epoch: 8 [600/800 (75%)]	Loss: 0.405456
Train Epoch: 8 [700/800 (88%)]	Loss: 0.587713

Test set: Average loss: 0.4954, Accuracy: 155/200 (77.500%)

Train Epoch: 9 [0/800 (0%)]	Loss: 0.445754
Train Epoch: 9 [100/800 (12%)]	Loss: 0.445473
Train Epoch: 9 [200/800 (25%)]	Loss: 0.474659
Train Epoch: 9 [300/800 (38%)]	Loss: 0.367472
Train Epoch: 9 [400/800 (50%)]	Loss: 0.447237
Train Epoch: 9 [500/800 (62%)]	Loss: 0.564817
Train Epoch: 9 [600/800 (75%)]	Loss: 0.490990
Train Epoch: 9 [700/800 (88%)]	Loss: 0.513888

Test set: Average loss: 0.4939, Accuracy: 153/200 (76.500%)

Train Epoch: 10 [0/800 (0%)]	Loss: 0.344564
Train Epoch: 10 [100/800 (12%)]	Loss: 0.182895
Train Epoch: 10 [200/800 (25%)]	Loss: 0.510447
Train Epoch: 10 [300/800 (38%)]	Loss: 0.656765
Train Epoch: 10 [400/800 (50%)]	Loss: 0.662577
Train Epoch: 10 [500/800 (62%)]	Loss: 0.552037
Train Epoch: 10 [600/800 (75%)]	Loss: 0.541802
Train Epoch: 10 [700/800 (88%)]	Loss: 0.297868

Test set: Average loss: 0.5122, Accuracy: 155/200 (77.500%)


 training DNN with  1000 data points and SGD lr=0.100000. 

Train Epoch: 1 [0/800 (0%)]	Loss: 0.702246
Train Epoch: 1 [100/800 (12%)]	Loss: 0.680417
Train Epoch: 1 [200/800 (25%)]	Loss: 0.738558
Train Epoch: 1 [300/800 (38%)]	Loss: 0.647689
Train Epoch: 1 [400/800 (50%)]	Loss: 0.435705
Train Epoch: 1 [500/800 (62%)]	Loss: 0.433396
Train Epoch: 1 [600/800 (75%)]	Loss: 0.504250
Train Epoch: 1 [700/800 (88%)]	Loss: 0.557903

Test set: Average loss: 0.5641, Accuracy: 142/200 (71.000%)

Train Epoch: 2 [0/800 (0%)]	Loss: 0.489483
Train Epoch: 2 [100/800 (12%)]	Loss: 0.553770
Train Epoch: 2 [200/800 (25%)]	Loss: 0.522039
Train Epoch: 2 [300/800 (38%)]	Loss: 0.674991
Train Epoch: 2 [400/800 (50%)]	Loss: 0.442521
Train Epoch: 2 [500/800 (62%)]	Loss: 0.477862
Train Epoch: 2 [600/800 (75%)]	Loss: 0.665187
Train Epoch: 2 [700/800 (88%)]	Loss: 0.965202

Test set: Average loss: 0.7319, Accuracy: 123/200 (61.500%)

Train Epoch: 3 [0/800 (0%)]	Loss: 0.719377
Train Epoch: 3 [100/800 (12%)]	Loss: 0.499032
Train Epoch: 3 [200/800 (25%)]	Loss: 0.653635
Train Epoch: 3 [300/800 (38%)]	Loss: 0.622866
Train Epoch: 3 [400/800 (50%)]	Loss: 0.615727
Train Epoch: 3 [500/800 (62%)]	Loss: 0.448293
Train Epoch: 3 [600/800 (75%)]	Loss: 0.953069
Train Epoch: 3 [700/800 (88%)]	Loss: 0.512476

Test set: Average loss: 0.5358, Accuracy: 146/200 (73.000%)

Train Epoch: 4 [0/800 (0%)]	Loss: 0.529396
Train Epoch: 4 [100/800 (12%)]	Loss: 0.400666
Train Epoch: 4 [200/800 (25%)]	Loss: 0.599574
Train Epoch: 4 [300/800 (38%)]	Loss: 0.577725
Train Epoch: 4 [400/800 (50%)]	Loss: 0.635720
Train Epoch: 4 [500/800 (62%)]	Loss: 0.634720
Train Epoch: 4 [600/800 (75%)]	Loss: 0.756374
Train Epoch: 4 [700/800 (88%)]	Loss: 1.011996

Test set: Average loss: 0.5952, Accuracy: 132/200 (66.000%)

Train Epoch: 5 [0/800 (0%)]	Loss: 0.830493
Train Epoch: 5 [100/800 (12%)]	Loss: 0.502611
Train Epoch: 5 [200/800 (25%)]	Loss: 0.730146
Train Epoch: 5 [300/800 (38%)]	Loss: 0.625743
Train Epoch: 5 [400/800 (50%)]	Loss: 0.264062
Train Epoch: 5 [500/800 (62%)]	Loss: 0.484053
Train Epoch: 5 [600/800 (75%)]	Loss: 0.861344
Train Epoch: 5 [700/800 (88%)]	Loss: 0.451043

Test set: Average loss: 0.5973, Accuracy: 142/200 (71.000%)

Train Epoch: 6 [0/800 (0%)]	Loss: 0.250868
Train Epoch: 6 [100/800 (12%)]	Loss: 0.459612
Train Epoch: 6 [200/800 (25%)]	Loss: 0.406168
Train Epoch: 6 [300/800 (38%)]	Loss: 0.551782
Train Epoch: 6 [400/800 (50%)]	Loss: 0.553747
Train Epoch: 6 [500/800 (62%)]	Loss: 0.554346
Train Epoch: 6 [600/800 (75%)]	Loss: 0.602257
Train Epoch: 6 [700/800 (88%)]	Loss: 0.317622

Test set: Average loss: 0.5605, Accuracy: 147/200 (73.500%)

Train Epoch: 7 [0/800 (0%)]	Loss: 0.515653
Train Epoch: 7 [100/800 (12%)]	Loss: 0.415217
Train Epoch: 7 [200/800 (25%)]	Loss: 0.647236
Train Epoch: 7 [300/800 (38%)]	Loss: 0.223138
Train Epoch: 7 [400/800 (50%)]	Loss: 0.409093
Train Epoch: 7 [500/800 (62%)]	Loss: 0.727628
Train Epoch: 7 [600/800 (75%)]	Loss: 1.045135
Train Epoch: 7 [700/800 (88%)]	Loss: 0.990046

Test set: Average loss: 0.5733, Accuracy: 135/200 (67.500%)

Train Epoch: 8 [0/800 (0%)]	Loss: 0.772774
Train Epoch: 8 [100/800 (12%)]	Loss: 0.306425
Train Epoch: 8 [200/800 (25%)]	Loss: 0.386095
Train Epoch: 8 [300/800 (38%)]	Loss: 0.432196
Train Epoch: 8 [400/800 (50%)]	Loss: 1.216796
Train Epoch: 8 [500/800 (62%)]	Loss: 0.568146
Train Epoch: 8 [600/800 (75%)]	Loss: 0.602302
Train Epoch: 8 [700/800 (88%)]	Loss: 0.442552

Test set: Average loss: 0.5602, Accuracy: 150/200 (75.000%)

Train Epoch: 9 [0/800 (0%)]	Loss: 0.417107
Train Epoch: 9 [100/800 (12%)]	Loss: 0.529822
Train Epoch: 9 [200/800 (25%)]	Loss: 0.277546
Train Epoch: 9 [300/800 (38%)]	Loss: 0.268979
Train Epoch: 9 [400/800 (50%)]	Loss: 0.325881
Train Epoch: 9 [500/800 (62%)]	Loss: 0.531899
Train Epoch: 9 [600/800 (75%)]	Loss: 0.508487
Train Epoch: 9 [700/800 (88%)]	Loss: 0.487037

Test set: Average loss: 0.6176, Accuracy: 126/200 (63.000%)

Train Epoch: 10 [0/800 (0%)]	Loss: 0.764035
Train Epoch: 10 [100/800 (12%)]	Loss: 0.644874
Train Epoch: 10 [200/800 (25%)]	Loss: 0.477930
Train Epoch: 10 [300/800 (38%)]	Loss: 0.346694
Train Epoch: 10 [400/800 (50%)]	Loss: 0.507348
Train Epoch: 10 [500/800 (62%)]	Loss: 0.702342
Train Epoch: 10 [600/800 (75%)]	Loss: 0.613372
Train Epoch: 10 [700/800 (88%)]	Loss: 0.572796

Test set: Average loss: 0.5932, Accuracy: 131/200 (65.500%)

Training on 8000 examples
Using both high and low level features
Testing on 2000 examples
Using both high and low level features

 training DNN with 10000 data points and SGD lr=0.000010. 

Train Epoch: 1 [0/8000 (0%)]	Loss: 0.727676
Train Epoch: 1 [1000/8000 (12%)]	Loss: 0.704320
Train Epoch: 1 [2000/8000 (25%)]	Loss: 0.710183
Train Epoch: 1 [3000/8000 (38%)]	Loss: 0.733221
Train Epoch: 1 [4000/8000 (50%)]	Loss: 0.723302
Train Epoch: 1 [5000/8000 (62%)]	Loss: 0.692780
Train Epoch: 1 [6000/8000 (75%)]	Loss: 0.721705
Train Epoch: 1 [7000/8000 (88%)]	Loss: 0.718693

Test set: Average loss: 0.7014, Accuracy: 879/2000 (43.950%)

Train Epoch: 2 [0/8000 (0%)]	Loss: 0.726935
Train Epoch: 2 [1000/8000 (12%)]	Loss: 0.687883
Train Epoch: 2 [2000/8000 (25%)]	Loss: 0.731870
Train Epoch: 2 [3000/8000 (38%)]	Loss: 0.698007
Train Epoch: 2 [4000/8000 (50%)]	Loss: 0.705002
Train Epoch: 2 [5000/8000 (62%)]	Loss: 0.719240
Train Epoch: 2 [6000/8000 (75%)]	Loss: 0.697965
Train Epoch: 2 [7000/8000 (88%)]	Loss: 0.710202

Test set: Average loss: 0.7013, Accuracy: 883/2000 (44.150%)

Train Epoch: 3 [0/8000 (0%)]	Loss: 0.722399
Train Epoch: 3 [1000/8000 (12%)]	Loss: 0.695973
Train Epoch: 3 [2000/8000 (25%)]	Loss: 0.709207
Train Epoch: 3 [3000/8000 (38%)]	Loss: 0.732915
Train Epoch: 3 [4000/8000 (50%)]	Loss: 0.690858
Train Epoch: 3 [5000/8000 (62%)]	Loss: 0.715846
Train Epoch: 3 [6000/8000 (75%)]	Loss: 0.711072
Train Epoch: 3 [7000/8000 (88%)]	Loss: 0.681372

Test set: Average loss: 0.7011, Accuracy: 883/2000 (44.150%)

Train Epoch: 4 [0/8000 (0%)]	Loss: 0.711699
Train Epoch: 4 [1000/8000 (12%)]	Loss: 0.701226
Train Epoch: 4 [2000/8000 (25%)]	Loss: 0.702945
Train Epoch: 4 [3000/8000 (38%)]	Loss: 0.671898
Train Epoch: 4 [4000/8000 (50%)]	Loss: 0.713768
Train Epoch: 4 [5000/8000 (62%)]	Loss: 0.707367
Train Epoch: 4 [6000/8000 (75%)]	Loss: 0.728022
Train Epoch: 4 [7000/8000 (88%)]	Loss: 0.703425

Test set: Average loss: 0.7010, Accuracy: 883/2000 (44.150%)

Train Epoch: 5 [0/8000 (0%)]	Loss: 0.709825
Train Epoch: 5 [1000/8000 (12%)]	Loss: 0.708488
Train Epoch: 5 [2000/8000 (25%)]	Loss: 0.676953
Train Epoch: 5 [3000/8000 (38%)]	Loss: 0.733793
Train Epoch: 5 [4000/8000 (50%)]	Loss: 0.748478
Train Epoch: 5 [5000/8000 (62%)]	Loss: 0.674785
Train Epoch: 5 [6000/8000 (75%)]	Loss: 0.725700
Train Epoch: 5 [7000/8000 (88%)]	Loss: 0.698330

Test set: Average loss: 0.7009, Accuracy: 885/2000 (44.250%)

Train Epoch: 6 [0/8000 (0%)]	Loss: 0.709079
Train Epoch: 6 [1000/8000 (12%)]	Loss: 0.696563
Train Epoch: 6 [2000/8000 (25%)]	Loss: 0.732473
Train Epoch: 6 [3000/8000 (38%)]	Loss: 0.698275
Train Epoch: 6 [4000/8000 (50%)]	Loss: 0.705673
Train Epoch: 6 [5000/8000 (62%)]	Loss: 0.714218
Train Epoch: 6 [6000/8000 (75%)]	Loss: 0.709984
Train Epoch: 6 [7000/8000 (88%)]	Loss: 0.711174

Test set: Average loss: 0.7008, Accuracy: 886/2000 (44.300%)

Train Epoch: 7 [0/8000 (0%)]	Loss: 0.706600
Train Epoch: 7 [1000/8000 (12%)]	Loss: 0.701822
Train Epoch: 7 [2000/8000 (25%)]	Loss: 0.732518
Train Epoch: 7 [3000/8000 (38%)]	Loss: 0.688266
Train Epoch: 7 [4000/8000 (50%)]	Loss: 0.702672
Train Epoch: 7 [5000/8000 (62%)]	Loss: 0.698141
Train Epoch: 7 [6000/8000 (75%)]	Loss: 0.706397
Train Epoch: 7 [7000/8000 (88%)]	Loss: 0.698923

Test set: Average loss: 0.7006, Accuracy: 887/2000 (44.350%)

Train Epoch: 8 [0/8000 (0%)]	Loss: 0.717546
Train Epoch: 8 [1000/8000 (12%)]	Loss: 0.702528
Train Epoch: 8 [2000/8000 (25%)]	Loss: 0.717072
Train Epoch: 8 [3000/8000 (38%)]	Loss: 0.704772
Train Epoch: 8 [4000/8000 (50%)]	Loss: 0.712914
Train Epoch: 8 [5000/8000 (62%)]	Loss: 0.699612
Train Epoch: 8 [6000/8000 (75%)]	Loss: 0.732635
Train Epoch: 8 [7000/8000 (88%)]	Loss: 0.670173

Test set: Average loss: 0.7005, Accuracy: 887/2000 (44.350%)

Train Epoch: 9 [0/8000 (0%)]	Loss: 0.705649
Train Epoch: 9 [1000/8000 (12%)]	Loss: 0.702978
Train Epoch: 9 [2000/8000 (25%)]	Loss: 0.706398
Train Epoch: 9 [3000/8000 (38%)]	Loss: 0.719252
Train Epoch: 9 [4000/8000 (50%)]	Loss: 0.715954
Train Epoch: 9 [5000/8000 (62%)]	Loss: 0.683632
Train Epoch: 9 [6000/8000 (75%)]	Loss: 0.708228
Train Epoch: 9 [7000/8000 (88%)]	Loss: 0.696788

Test set: Average loss: 0.7004, Accuracy: 892/2000 (44.600%)

Train Epoch: 10 [0/8000 (0%)]	Loss: 0.700282
Train Epoch: 10 [1000/8000 (12%)]	Loss: 0.724479
Train Epoch: 10 [2000/8000 (25%)]	Loss: 0.709506
Train Epoch: 10 [3000/8000 (38%)]	Loss: 0.708790
Train Epoch: 10 [4000/8000 (50%)]	Loss: 0.703068
Train Epoch: 10 [5000/8000 (62%)]	Loss: 0.713272
Train Epoch: 10 [6000/8000 (75%)]	Loss: 0.716486
Train Epoch: 10 [7000/8000 (88%)]	Loss: 0.688014

Test set: Average loss: 0.7003, Accuracy: 893/2000 (44.650%)


 training DNN with 10000 data points and SGD lr=0.000100. 

Train Epoch: 1 [0/8000 (0%)]	Loss: 0.710169
Train Epoch: 1 [1000/8000 (12%)]	Loss: 0.691915
Train Epoch: 1 [2000/8000 (25%)]	Loss: 0.691650
Train Epoch: 1 [3000/8000 (38%)]	Loss: 0.713645
Train Epoch: 1 [4000/8000 (50%)]	Loss: 0.727196
Train Epoch: 1 [5000/8000 (62%)]	Loss: 0.707788
Train Epoch: 1 [6000/8000 (75%)]	Loss: 0.710902
Train Epoch: 1 [7000/8000 (88%)]	Loss: 0.692166

Test set: Average loss: 0.6919, Accuracy: 1082/2000 (54.100%)

Train Epoch: 2 [0/8000 (0%)]	Loss: 0.715787
Train Epoch: 2 [1000/8000 (12%)]	Loss: 0.716278
Train Epoch: 2 [2000/8000 (25%)]	Loss: 0.681749
Train Epoch: 2 [3000/8000 (38%)]	Loss: 0.684479
Train Epoch: 2 [4000/8000 (50%)]	Loss: 0.690353
Train Epoch: 2 [5000/8000 (62%)]	Loss: 0.686106
Train Epoch: 2 [6000/8000 (75%)]	Loss: 0.694722
Train Epoch: 2 [7000/8000 (88%)]	Loss: 0.683836

Test set: Average loss: 0.6904, Accuracy: 1127/2000 (56.350%)

Train Epoch: 3 [0/8000 (0%)]	Loss: 0.684679
Train Epoch: 3 [1000/8000 (12%)]	Loss: 0.674189
Train Epoch: 3 [2000/8000 (25%)]	Loss: 0.700893
Train Epoch: 3 [3000/8000 (38%)]	Loss: 0.701272
Train Epoch: 3 [4000/8000 (50%)]	Loss: 0.692683
Train Epoch: 3 [5000/8000 (62%)]	Loss: 0.698199
Train Epoch: 3 [6000/8000 (75%)]	Loss: 0.703976
Train Epoch: 3 [7000/8000 (88%)]	Loss: 0.707629

Test set: Average loss: 0.6890, Accuracy: 1151/2000 (57.550%)

Train Epoch: 4 [0/8000 (0%)]	Loss: 0.703755
Train Epoch: 4 [1000/8000 (12%)]	Loss: 0.692734
Train Epoch: 4 [2000/8000 (25%)]	Loss: 0.717795
Train Epoch: 4 [3000/8000 (38%)]	Loss: 0.711542
Train Epoch: 4 [4000/8000 (50%)]	Loss: 0.682708
Train Epoch: 4 [5000/8000 (62%)]	Loss: 0.697612
Train Epoch: 4 [6000/8000 (75%)]	Loss: 0.709732
Train Epoch: 4 [7000/8000 (88%)]	Loss: 0.685663

Test set: Average loss: 0.6877, Accuracy: 1182/2000 (59.100%)

Train Epoch: 5 [0/8000 (0%)]	Loss: 0.699202
Train Epoch: 5 [1000/8000 (12%)]	Loss: 0.681597
Train Epoch: 5 [2000/8000 (25%)]	Loss: 0.703145
Train Epoch: 5 [3000/8000 (38%)]	Loss: 0.681526
Train Epoch: 5 [4000/8000 (50%)]	Loss: 0.670278
Train Epoch: 5 [5000/8000 (62%)]	Loss: 0.700635
Train Epoch: 5 [6000/8000 (75%)]	Loss: 0.687604
Train Epoch: 5 [7000/8000 (88%)]	Loss: 0.685772

Test set: Average loss: 0.6864, Accuracy: 1210/2000 (60.500%)

Train Epoch: 6 [0/8000 (0%)]	Loss: 0.690343
Train Epoch: 6 [1000/8000 (12%)]	Loss: 0.687278
Train Epoch: 6 [2000/8000 (25%)]	Loss: 0.675960
Train Epoch: 6 [3000/8000 (38%)]	Loss: 0.705044
Train Epoch: 6 [4000/8000 (50%)]	Loss: 0.703282
Train Epoch: 6 [5000/8000 (62%)]	Loss: 0.703023
Train Epoch: 6 [6000/8000 (75%)]	Loss: 0.687314
Train Epoch: 6 [7000/8000 (88%)]	Loss: 0.703106

Test set: Average loss: 0.6852, Accuracy: 1235/2000 (61.750%)

Train Epoch: 7 [0/8000 (0%)]	Loss: 0.671762
Train Epoch: 7 [1000/8000 (12%)]	Loss: 0.701850
Train Epoch: 7 [2000/8000 (25%)]	Loss: 0.707641
Train Epoch: 7 [3000/8000 (38%)]	Loss: 0.703546
Train Epoch: 7 [4000/8000 (50%)]	Loss: 0.711041
Train Epoch: 7 [5000/8000 (62%)]	Loss: 0.693455
Train Epoch: 7 [6000/8000 (75%)]	Loss: 0.696086
Train Epoch: 7 [7000/8000 (88%)]	Loss: 0.686095

Test set: Average loss: 0.6841, Accuracy: 1244/2000 (62.200%)

Train Epoch: 8 [0/8000 (0%)]	Loss: 0.686531
Train Epoch: 8 [1000/8000 (12%)]	Loss: 0.677912
Train Epoch: 8 [2000/8000 (25%)]	Loss: 0.686239
Train Epoch: 8 [3000/8000 (38%)]	Loss: 0.704038
Train Epoch: 8 [4000/8000 (50%)]	Loss: 0.697486
Train Epoch: 8 [5000/8000 (62%)]	Loss: 0.677249
Train Epoch: 8 [6000/8000 (75%)]	Loss: 0.680064
Train Epoch: 8 [7000/8000 (88%)]	Loss: 0.680018

Test set: Average loss: 0.6830, Accuracy: 1259/2000 (62.950%)

Train Epoch: 9 [0/8000 (0%)]	Loss: 0.681513
Train Epoch: 9 [1000/8000 (12%)]	Loss: 0.676095
Train Epoch: 9 [2000/8000 (25%)]	Loss: 0.683777
Train Epoch: 9 [3000/8000 (38%)]	Loss: 0.691705
Train Epoch: 9 [4000/8000 (50%)]	Loss: 0.696020
Train Epoch: 9 [5000/8000 (62%)]	Loss: 0.684099
Train Epoch: 9 [6000/8000 (75%)]	Loss: 0.695634
Train Epoch: 9 [7000/8000 (88%)]	Loss: 0.688131

Test set: Average loss: 0.6819, Accuracy: 1281/2000 (64.050%)

Train Epoch: 10 [0/8000 (0%)]	Loss: 0.674268
Train Epoch: 10 [1000/8000 (12%)]	Loss: 0.674218
Train Epoch: 10 [2000/8000 (25%)]	Loss: 0.701858
Train Epoch: 10 [3000/8000 (38%)]	Loss: 0.675874
Train Epoch: 10 [4000/8000 (50%)]	Loss: 0.702081
Train Epoch: 10 [5000/8000 (62%)]	Loss: 0.703176
Train Epoch: 10 [6000/8000 (75%)]	Loss: 0.692648
Train Epoch: 10 [7000/8000 (88%)]	Loss: 0.669730

Test set: Average loss: 0.6808, Accuracy: 1301/2000 (65.050%)


 training DNN with 10000 data points and SGD lr=0.001000. 

Train Epoch: 1 [0/8000 (0%)]	Loss: 0.748844
Train Epoch: 1 [1000/8000 (12%)]	Loss: 0.710549
Train Epoch: 1 [2000/8000 (25%)]	Loss: 0.725623
Train Epoch: 1 [3000/8000 (38%)]	Loss: 0.720901
Train Epoch: 1 [4000/8000 (50%)]	Loss: 0.711438
Train Epoch: 1 [5000/8000 (62%)]	Loss: 0.724673
Train Epoch: 1 [6000/8000 (75%)]	Loss: 0.689648
Train Epoch: 1 [7000/8000 (88%)]	Loss: 0.719881

Test set: Average loss: 0.6967, Accuracy: 937/2000 (46.850%)

Train Epoch: 2 [0/8000 (0%)]	Loss: 0.722610
Train Epoch: 2 [1000/8000 (12%)]	Loss: 0.712821
Train Epoch: 2 [2000/8000 (25%)]	Loss: 0.709567
Train Epoch: 2 [3000/8000 (38%)]	Loss: 0.705417
Train Epoch: 2 [4000/8000 (50%)]	Loss: 0.695027
Train Epoch: 2 [5000/8000 (62%)]	Loss: 0.707301
Train Epoch: 2 [6000/8000 (75%)]	Loss: 0.713473
Train Epoch: 2 [7000/8000 (88%)]	Loss: 0.702530

Test set: Average loss: 0.6841, Accuracy: 1068/2000 (53.400%)

Train Epoch: 3 [0/8000 (0%)]	Loss: 0.697902
Train Epoch: 3 [1000/8000 (12%)]	Loss: 0.687280
Train Epoch: 3 [2000/8000 (25%)]	Loss: 0.703073
Train Epoch: 3 [3000/8000 (38%)]	Loss: 0.695030
Train Epoch: 3 [4000/8000 (50%)]	Loss: 0.687740
Train Epoch: 3 [5000/8000 (62%)]	Loss: 0.692897
Train Epoch: 3 [6000/8000 (75%)]	Loss: 0.695042
Train Epoch: 3 [7000/8000 (88%)]	Loss: 0.673611

Test set: Average loss: 0.6742, Accuracy: 1358/2000 (67.900%)

Train Epoch: 4 [0/8000 (0%)]	Loss: 0.679266
Train Epoch: 4 [1000/8000 (12%)]	Loss: 0.688101
Train Epoch: 4 [2000/8000 (25%)]	Loss: 0.663009
Train Epoch: 4 [3000/8000 (38%)]	Loss: 0.685576
Train Epoch: 4 [4000/8000 (50%)]	Loss: 0.698362
Train Epoch: 4 [5000/8000 (62%)]	Loss: 0.681344
Train Epoch: 4 [6000/8000 (75%)]	Loss: 0.656040
Train Epoch: 4 [7000/8000 (88%)]	Loss: 0.668664

Test set: Average loss: 0.6644, Accuracy: 1437/2000 (71.850%)

Train Epoch: 5 [0/8000 (0%)]	Loss: 0.657408
Train Epoch: 5 [1000/8000 (12%)]	Loss: 0.652054
Train Epoch: 5 [2000/8000 (25%)]	Loss: 0.684324
Train Epoch: 5 [3000/8000 (38%)]	Loss: 0.684844
Train Epoch: 5 [4000/8000 (50%)]	Loss: 0.661951
Train Epoch: 5 [5000/8000 (62%)]	Loss: 0.682703
Train Epoch: 5 [6000/8000 (75%)]	Loss: 0.670805
Train Epoch: 5 [7000/8000 (88%)]	Loss: 0.667759

Test set: Average loss: 0.6545, Accuracy: 1462/2000 (73.100%)

Train Epoch: 6 [0/8000 (0%)]	Loss: 0.663250
Train Epoch: 6 [1000/8000 (12%)]	Loss: 0.672179
Train Epoch: 6 [2000/8000 (25%)]	Loss: 0.669661
Train Epoch: 6 [3000/8000 (38%)]	Loss: 0.673492
Train Epoch: 6 [4000/8000 (50%)]	Loss: 0.646684
Train Epoch: 6 [5000/8000 (62%)]	Loss: 0.650809
Train Epoch: 6 [6000/8000 (75%)]	Loss: 0.650157
Train Epoch: 6 [7000/8000 (88%)]	Loss: 0.665025

Test set: Average loss: 0.6449, Accuracy: 1474/2000 (73.700%)

Train Epoch: 7 [0/8000 (0%)]	Loss: 0.668964
Train Epoch: 7 [1000/8000 (12%)]	Loss: 0.650208
Train Epoch: 7 [2000/8000 (25%)]	Loss: 0.669286
Train Epoch: 7 [3000/8000 (38%)]	Loss: 0.637196
Train Epoch: 7 [4000/8000 (50%)]	Loss: 0.680341
Train Epoch: 7 [5000/8000 (62%)]	Loss: 0.641309
Train Epoch: 7 [6000/8000 (75%)]	Loss: 0.651198
Train Epoch: 7 [7000/8000 (88%)]	Loss: 0.660376

Test set: Average loss: 0.6362, Accuracy: 1476/2000 (73.800%)

Train Epoch: 8 [0/8000 (0%)]	Loss: 0.661032
Train Epoch: 8 [1000/8000 (12%)]	Loss: 0.651090
Train Epoch: 8 [2000/8000 (25%)]	Loss: 0.631498
Train Epoch: 8 [3000/8000 (38%)]	Loss: 0.666181
Train Epoch: 8 [4000/8000 (50%)]	Loss: 0.670746
Train Epoch: 8 [5000/8000 (62%)]	Loss: 0.652429
Train Epoch: 8 [6000/8000 (75%)]	Loss: 0.623989
Train Epoch: 8 [7000/8000 (88%)]	Loss: 0.647921

Test set: Average loss: 0.6275, Accuracy: 1489/2000 (74.450%)

Train Epoch: 9 [0/8000 (0%)]	Loss: 0.638977
Train Epoch: 9 [1000/8000 (12%)]	Loss: 0.632178
Train Epoch: 9 [2000/8000 (25%)]	Loss: 0.628478
Train Epoch: 9 [3000/8000 (38%)]	Loss: 0.645219
Train Epoch: 9 [4000/8000 (50%)]	Loss: 0.634084
Train Epoch: 9 [5000/8000 (62%)]	Loss: 0.660404
Train Epoch: 9 [6000/8000 (75%)]	Loss: 0.637642
Train Epoch: 9 [7000/8000 (88%)]	Loss: 0.629800

Test set: Average loss: 0.6189, Accuracy: 1478/2000 (73.900%)

Train Epoch: 10 [0/8000 (0%)]	Loss: 0.626010
Train Epoch: 10 [1000/8000 (12%)]	Loss: 0.640649
Train Epoch: 10 [2000/8000 (25%)]	Loss: 0.661770
Train Epoch: 10 [3000/8000 (38%)]	Loss: 0.641002
Train Epoch: 10 [4000/8000 (50%)]	Loss: 0.611413
Train Epoch: 10 [5000/8000 (62%)]	Loss: 0.646812
Train Epoch: 10 [6000/8000 (75%)]	Loss: 0.634771
Train Epoch: 10 [7000/8000 (88%)]	Loss: 0.637156

Test set: Average loss: 0.6100, Accuracy: 1491/2000 (74.550%)


 training DNN with 10000 data points and SGD lr=0.010000. 

Train Epoch: 1 [0/8000 (0%)]	Loss: 0.699168
Train Epoch: 1 [1000/8000 (12%)]	Loss: 0.697183
Train Epoch: 1 [2000/8000 (25%)]	Loss: 0.673701
Train Epoch: 1 [3000/8000 (38%)]	Loss: 0.652901
Train Epoch: 1 [4000/8000 (50%)]	Loss: 0.687900
Train Epoch: 1 [5000/8000 (62%)]	Loss: 0.663231
Train Epoch: 1 [6000/8000 (75%)]	Loss: 0.657198
Train Epoch: 1 [7000/8000 (88%)]	Loss: 0.618428

Test set: Average loss: 0.6132, Accuracy: 1506/2000 (75.300%)

Train Epoch: 2 [0/8000 (0%)]	Loss: 0.604715
Train Epoch: 2 [1000/8000 (12%)]	Loss: 0.611860
Train Epoch: 2 [2000/8000 (25%)]	Loss: 0.609607
Train Epoch: 2 [3000/8000 (38%)]	Loss: 0.609965
Train Epoch: 2 [4000/8000 (50%)]	Loss: 0.575748
Train Epoch: 2 [5000/8000 (62%)]	Loss: 0.589488
Train Epoch: 2 [6000/8000 (75%)]	Loss: 0.623795
Train Epoch: 2 [7000/8000 (88%)]	Loss: 0.580438

Test set: Average loss: 0.5488, Accuracy: 1526/2000 (76.300%)

Train Epoch: 3 [0/8000 (0%)]	Loss: 0.560898
Train Epoch: 3 [1000/8000 (12%)]	Loss: 0.571261
Train Epoch: 3 [2000/8000 (25%)]	Loss: 0.540140
Train Epoch: 3 [3000/8000 (38%)]	Loss: 0.611341
Train Epoch: 3 [4000/8000 (50%)]	Loss: 0.604470
Train Epoch: 3 [5000/8000 (62%)]	Loss: 0.547775
Train Epoch: 3 [6000/8000 (75%)]	Loss: 0.620242
Train Epoch: 3 [7000/8000 (88%)]	Loss: 0.551052

Test set: Average loss: 0.5134, Accuracy: 1536/2000 (76.800%)

Train Epoch: 4 [0/8000 (0%)]	Loss: 0.504537
Train Epoch: 4 [1000/8000 (12%)]	Loss: 0.512580
Train Epoch: 4 [2000/8000 (25%)]	Loss: 0.545692
Train Epoch: 4 [3000/8000 (38%)]	Loss: 0.534161
Train Epoch: 4 [4000/8000 (50%)]	Loss: 0.488619
Train Epoch: 4 [5000/8000 (62%)]	Loss: 0.592135
Train Epoch: 4 [6000/8000 (75%)]	Loss: 0.550200
Train Epoch: 4 [7000/8000 (88%)]	Loss: 0.520150

Test set: Average loss: 0.4982, Accuracy: 1529/2000 (76.450%)

Train Epoch: 5 [0/8000 (0%)]	Loss: 0.545439
Train Epoch: 5 [1000/8000 (12%)]	Loss: 0.553422
Train Epoch: 5 [2000/8000 (25%)]	Loss: 0.530373
Train Epoch: 5 [3000/8000 (38%)]	Loss: 0.522281
Train Epoch: 5 [4000/8000 (50%)]	Loss: 0.566266
Train Epoch: 5 [5000/8000 (62%)]	Loss: 0.526884
Train Epoch: 5 [6000/8000 (75%)]	Loss: 0.549719
Train Epoch: 5 [7000/8000 (88%)]	Loss: 0.517988

Test set: Average loss: 0.4864, Accuracy: 1529/2000 (76.450%)

Train Epoch: 6 [0/8000 (0%)]	Loss: 0.480552
Train Epoch: 6 [1000/8000 (12%)]	Loss: 0.573093
Train Epoch: 6 [2000/8000 (25%)]	Loss: 0.575137
Train Epoch: 6 [3000/8000 (38%)]	Loss: 0.569813
Train Epoch: 6 [4000/8000 (50%)]	Loss: 0.444771
Train Epoch: 6 [5000/8000 (62%)]	Loss: 0.497351
Train Epoch: 6 [6000/8000 (75%)]	Loss: 0.527475
Train Epoch: 6 [7000/8000 (88%)]	Loss: 0.515924

Test set: Average loss: 0.4803, Accuracy: 1529/2000 (76.450%)

Train Epoch: 7 [0/8000 (0%)]	Loss: 0.497473
Train Epoch: 7 [1000/8000 (12%)]	Loss: 0.530438
Train Epoch: 7 [2000/8000 (25%)]	Loss: 0.514741
Train Epoch: 7 [3000/8000 (38%)]	Loss: 0.527368
Train Epoch: 7 [4000/8000 (50%)]	Loss: 0.482035
Train Epoch: 7 [5000/8000 (62%)]	Loss: 0.485020
Train Epoch: 7 [6000/8000 (75%)]	Loss: 0.440541
Train Epoch: 7 [7000/8000 (88%)]	Loss: 0.483561

Test set: Average loss: 0.4723, Accuracy: 1549/2000 (77.450%)

Train Epoch: 8 [0/8000 (0%)]	Loss: 0.462284
Train Epoch: 8 [1000/8000 (12%)]	Loss: 0.585840
Train Epoch: 8 [2000/8000 (25%)]	Loss: 0.454685
Train Epoch: 8 [3000/8000 (38%)]	Loss: 0.497827
Train Epoch: 8 [4000/8000 (50%)]	Loss: 0.532175
Train Epoch: 8 [5000/8000 (62%)]	Loss: 0.533099
Train Epoch: 8 [6000/8000 (75%)]	Loss: 0.513271
Train Epoch: 8 [7000/8000 (88%)]	Loss: 0.471093

Test set: Average loss: 0.4704, Accuracy: 1545/2000 (77.250%)

Train Epoch: 9 [0/8000 (0%)]	Loss: 0.559074
Train Epoch: 9 [1000/8000 (12%)]	Loss: 0.408493
Train Epoch: 9 [2000/8000 (25%)]	Loss: 0.576147
Train Epoch: 9 [3000/8000 (38%)]	Loss: 0.516105
Train Epoch: 9 [4000/8000 (50%)]	Loss: 0.499826
Train Epoch: 9 [5000/8000 (62%)]	Loss: 0.421751
Train Epoch: 9 [6000/8000 (75%)]	Loss: 0.495686
Train Epoch: 9 [7000/8000 (88%)]	Loss: 0.434827

Test set: Average loss: 0.4665, Accuracy: 1558/2000 (77.900%)

Train Epoch: 10 [0/8000 (0%)]	Loss: 0.477638
Train Epoch: 10 [1000/8000 (12%)]	Loss: 0.602242
Train Epoch: 10 [2000/8000 (25%)]	Loss: 0.509296
Train Epoch: 10 [3000/8000 (38%)]	Loss: 0.489183
Train Epoch: 10 [4000/8000 (50%)]	Loss: 0.520043
Train Epoch: 10 [5000/8000 (62%)]	Loss: 0.451816
Train Epoch: 10 [6000/8000 (75%)]	Loss: 0.499089
Train Epoch: 10 [7000/8000 (88%)]	Loss: 0.471891

Test set: Average loss: 0.4686, Accuracy: 1541/2000 (77.050%)


 training DNN with 10000 data points and SGD lr=0.100000. 

Train Epoch: 1 [0/8000 (0%)]	Loss: 0.685625
Train Epoch: 1 [1000/8000 (12%)]	Loss: 0.610574
Train Epoch: 1 [2000/8000 (25%)]	Loss: 0.605155
Train Epoch: 1 [3000/8000 (38%)]	Loss: 0.506837
Train Epoch: 1 [4000/8000 (50%)]	Loss: 0.513233
Train Epoch: 1 [5000/8000 (62%)]	Loss: 0.503912
Train Epoch: 1 [6000/8000 (75%)]	Loss: 0.533501
Train Epoch: 1 [7000/8000 (88%)]	Loss: 0.496268

Test set: Average loss: 0.4800, Accuracy: 1534/2000 (76.700%)

Train Epoch: 2 [0/8000 (0%)]	Loss: 0.466775
Train Epoch: 2 [1000/8000 (12%)]	Loss: 0.497168
Train Epoch: 2 [2000/8000 (25%)]	Loss: 0.471591
Train Epoch: 2 [3000/8000 (38%)]	Loss: 0.448215
Train Epoch: 2 [4000/8000 (50%)]	Loss: 0.502071
Train Epoch: 2 [5000/8000 (62%)]	Loss: 0.412387
Train Epoch: 2 [6000/8000 (75%)]	Loss: 0.621073
Train Epoch: 2 [7000/8000 (88%)]	Loss: 0.429254

Test set: Average loss: 0.4607, Accuracy: 1560/2000 (78.000%)

Train Epoch: 3 [0/8000 (0%)]	Loss: 0.383824
Train Epoch: 3 [1000/8000 (12%)]	Loss: 0.477260
Train Epoch: 3 [2000/8000 (25%)]	Loss: 0.541657
Train Epoch: 3 [3000/8000 (38%)]	Loss: 0.419515
Train Epoch: 3 [4000/8000 (50%)]	Loss: 0.534225
Train Epoch: 3 [5000/8000 (62%)]	Loss: 0.455988
Train Epoch: 3 [6000/8000 (75%)]	Loss: 0.336730
Train Epoch: 3 [7000/8000 (88%)]	Loss: 0.445602

Test set: Average loss: 0.4537, Accuracy: 1578/2000 (78.900%)

Train Epoch: 4 [0/8000 (0%)]	Loss: 0.428422
Train Epoch: 4 [1000/8000 (12%)]	Loss: 0.546706
Train Epoch: 4 [2000/8000 (25%)]	Loss: 0.538695
Train Epoch: 4 [3000/8000 (38%)]	Loss: 0.472345
Train Epoch: 4 [4000/8000 (50%)]	Loss: 0.527527
Train Epoch: 4 [5000/8000 (62%)]	Loss: 0.572210
Train Epoch: 4 [6000/8000 (75%)]	Loss: 0.393512
Train Epoch: 4 [7000/8000 (88%)]	Loss: 0.508052

Test set: Average loss: 0.4541, Accuracy: 1578/2000 (78.900%)

Train Epoch: 5 [0/8000 (0%)]	Loss: 0.498704
Train Epoch: 5 [1000/8000 (12%)]	Loss: 0.437050
Train Epoch: 5 [2000/8000 (25%)]	Loss: 0.484379
Train Epoch: 5 [3000/8000 (38%)]	Loss: 0.521299
Train Epoch: 5 [4000/8000 (50%)]	Loss: 0.464109
Train Epoch: 5 [5000/8000 (62%)]	Loss: 0.439022
Train Epoch: 5 [6000/8000 (75%)]	Loss: 0.469260
Train Epoch: 5 [7000/8000 (88%)]	Loss: 0.493705

Test set: Average loss: 0.4709, Accuracy: 1565/2000 (78.250%)

Train Epoch: 6 [0/8000 (0%)]	Loss: 0.531487
Train Epoch: 6 [1000/8000 (12%)]	Loss: 0.522983
Train Epoch: 6 [2000/8000 (25%)]	Loss: 0.504122
Train Epoch: 6 [3000/8000 (38%)]	Loss: 0.401215
Train Epoch: 6 [4000/8000 (50%)]	Loss: 0.449189
Train Epoch: 6 [5000/8000 (62%)]	Loss: 0.418882
Train Epoch: 6 [6000/8000 (75%)]	Loss: 0.461273
Train Epoch: 6 [7000/8000 (88%)]	Loss: 0.414933

Test set: Average loss: 0.4688, Accuracy: 1552/2000 (77.600%)

Train Epoch: 7 [0/8000 (0%)]	Loss: 0.515866
Train Epoch: 7 [1000/8000 (12%)]	Loss: 0.434252
Train Epoch: 7 [2000/8000 (25%)]	Loss: 0.464060
Train Epoch: 7 [3000/8000 (38%)]	Loss: 0.471080
Train Epoch: 7 [4000/8000 (50%)]	Loss: 0.492378
Train Epoch: 7 [5000/8000 (62%)]	Loss: 0.491037
Train Epoch: 7 [6000/8000 (75%)]	Loss: 0.393833
Train Epoch: 7 [7000/8000 (88%)]	Loss: 0.436135

Test set: Average loss: 0.4558, Accuracy: 1570/2000 (78.500%)

Train Epoch: 8 [0/8000 (0%)]	Loss: 0.412632
Train Epoch: 8 [1000/8000 (12%)]	Loss: 0.499017
Train Epoch: 8 [2000/8000 (25%)]	Loss: 0.443011
Train Epoch: 8 [3000/8000 (38%)]	Loss: 0.516305
Train Epoch: 8 [4000/8000 (50%)]	Loss: 0.513990
Train Epoch: 8 [5000/8000 (62%)]	Loss: 0.491332
Train Epoch: 8 [6000/8000 (75%)]	Loss: 0.474468
Train Epoch: 8 [7000/8000 (88%)]	Loss: 0.476098

Test set: Average loss: 0.4600, Accuracy: 1563/2000 (78.150%)

Train Epoch: 9 [0/8000 (0%)]	Loss: 0.476434
Train Epoch: 9 [1000/8000 (12%)]	Loss: 0.396529
Train Epoch: 9 [2000/8000 (25%)]	Loss: 0.454722
Train Epoch: 9 [3000/8000 (38%)]	Loss: 0.483790
Train Epoch: 9 [4000/8000 (50%)]	Loss: 0.400906
Train Epoch: 9 [5000/8000 (62%)]	Loss: 0.371000
Train Epoch: 9 [6000/8000 (75%)]	Loss: 0.377229
Train Epoch: 9 [7000/8000 (88%)]	Loss: 0.417167

Test set: Average loss: 0.4647, Accuracy: 1553/2000 (77.650%)

Train Epoch: 10 [0/8000 (0%)]	Loss: 0.470852
Train Epoch: 10 [1000/8000 (12%)]	Loss: 0.362817
Train Epoch: 10 [2000/8000 (25%)]	Loss: 0.473564
Train Epoch: 10 [3000/8000 (38%)]	Loss: 0.456163
Train Epoch: 10 [4000/8000 (50%)]	Loss: 0.503372
Train Epoch: 10 [5000/8000 (62%)]	Loss: 0.534373
Train Epoch: 10 [6000/8000 (75%)]	Loss: 0.437734
Train Epoch: 10 [7000/8000 (88%)]	Loss: 0.540555

Test set: Average loss: 0.4577, Accuracy: 1565/2000 (78.250%)

Training on 80000 examples
Using both high and low level features
Testing on 20000 examples
Using both high and low level features

 training DNN with 100000 data points and SGD lr=0.000010. 

Train Epoch: 1 [0/80000 (0%)]	Loss: 0.709857
Train Epoch: 1 [10000/80000 (12%)]	Loss: 0.699108
Train Epoch: 1 [20000/80000 (25%)]	Loss: 0.705935
Train Epoch: 1 [30000/80000 (38%)]	Loss: 0.707634
Train Epoch: 1 [40000/80000 (50%)]	Loss: 0.704721
Train Epoch: 1 [50000/80000 (62%)]	Loss: 0.701020
Train Epoch: 1 [60000/80000 (75%)]	Loss: 0.701019
Train Epoch: 1 [70000/80000 (88%)]	Loss: 0.700481

Test set: Average loss: 0.7008, Accuracy: 8467/20000 (42.335%)

Train Epoch: 2 [0/80000 (0%)]	Loss: 0.704691
Train Epoch: 2 [10000/80000 (12%)]	Loss: 0.705849
Train Epoch: 2 [20000/80000 (25%)]	Loss: 0.702281
Train Epoch: 2 [30000/80000 (38%)]	Loss: 0.702284
Train Epoch: 2 [40000/80000 (50%)]	Loss: 0.706607
Train Epoch: 2 [50000/80000 (62%)]	Loss: 0.700778
Train Epoch: 2 [60000/80000 (75%)]	Loss: 0.706767
Train Epoch: 2 [70000/80000 (88%)]	Loss: 0.697481

Test set: Average loss: 0.7007, Accuracy: 8471/20000 (42.355%)

Train Epoch: 3 [0/80000 (0%)]	Loss: 0.702649
Train Epoch: 3 [10000/80000 (12%)]	Loss: 0.699560
Train Epoch: 3 [20000/80000 (25%)]	Loss: 0.701000
Train Epoch: 3 [30000/80000 (38%)]	Loss: 0.697289
Train Epoch: 3 [40000/80000 (50%)]	Loss: 0.698617
Train Epoch: 3 [50000/80000 (62%)]	Loss: 0.699536
Train Epoch: 3 [60000/80000 (75%)]	Loss: 0.701737
Train Epoch: 3 [70000/80000 (88%)]	Loss: 0.707562

Test set: Average loss: 0.7005, Accuracy: 8478/20000 (42.390%)

Train Epoch: 4 [0/80000 (0%)]	Loss: 0.702440
Train Epoch: 4 [10000/80000 (12%)]	Loss: 0.701264
Train Epoch: 4 [20000/80000 (25%)]	Loss: 0.706735
Train Epoch: 4 [30000/80000 (38%)]	Loss: 0.705587
Train Epoch: 4 [40000/80000 (50%)]	Loss: 0.699598
Train Epoch: 4 [50000/80000 (62%)]	Loss: 0.704237
Train Epoch: 4 [60000/80000 (75%)]	Loss: 0.707695
Train Epoch: 4 [70000/80000 (88%)]	Loss: 0.704129

Test set: Average loss: 0.7004, Accuracy: 8493/20000 (42.465%)

Train Epoch: 5 [0/80000 (0%)]	Loss: 0.703403
Train Epoch: 5 [10000/80000 (12%)]	Loss: 0.704349
Train Epoch: 5 [20000/80000 (25%)]	Loss: 0.700892
Train Epoch: 5 [30000/80000 (38%)]	Loss: 0.702444
Train Epoch: 5 [40000/80000 (50%)]	Loss: 0.707312
Train Epoch: 5 [50000/80000 (62%)]	Loss: 0.702573
Train Epoch: 5 [60000/80000 (75%)]	Loss: 0.694193
Train Epoch: 5 [70000/80000 (88%)]	Loss: 0.707019

Test set: Average loss: 0.7003, Accuracy: 8487/20000 (42.435%)

Train Epoch: 6 [0/80000 (0%)]	Loss: 0.699891
Train Epoch: 6 [10000/80000 (12%)]	Loss: 0.700422
Train Epoch: 6 [20000/80000 (25%)]	Loss: 0.703033
Train Epoch: 6 [30000/80000 (38%)]	Loss: 0.700780
Train Epoch: 6 [40000/80000 (50%)]	Loss: 0.698106
Train Epoch: 6 [50000/80000 (62%)]	Loss: 0.704318
Train Epoch: 6 [60000/80000 (75%)]	Loss: 0.702799
Train Epoch: 6 [70000/80000 (88%)]	Loss: 0.705650

Test set: Average loss: 0.7001, Accuracy: 8488/20000 (42.440%)

Train Epoch: 7 [0/80000 (0%)]	Loss: 0.697022
Train Epoch: 7 [10000/80000 (12%)]	Loss: 0.706780
Train Epoch: 7 [20000/80000 (25%)]	Loss: 0.703789
Train Epoch: 7 [30000/80000 (38%)]	Loss: 0.695989
Train Epoch: 7 [40000/80000 (50%)]	Loss: 0.703647
Train Epoch: 7 [50000/80000 (62%)]	Loss: 0.699872
Train Epoch: 7 [60000/80000 (75%)]	Loss: 0.696702
Train Epoch: 7 [70000/80000 (88%)]	Loss: 0.695905

Test set: Average loss: 0.7000, Accuracy: 8502/20000 (42.510%)

Train Epoch: 8 [0/80000 (0%)]	Loss: 0.704256
Train Epoch: 8 [10000/80000 (12%)]	Loss: 0.705376
Train Epoch: 8 [20000/80000 (25%)]	Loss: 0.706469
Train Epoch: 8 [30000/80000 (38%)]	Loss: 0.701247
Train Epoch: 8 [40000/80000 (50%)]	Loss: 0.700315
Train Epoch: 8 [50000/80000 (62%)]	Loss: 0.703419
Train Epoch: 8 [60000/80000 (75%)]	Loss: 0.694629
Train Epoch: 8 [70000/80000 (88%)]	Loss: 0.704856

Test set: Average loss: 0.6999, Accuracy: 8523/20000 (42.615%)

Train Epoch: 9 [0/80000 (0%)]	Loss: 0.704544
Train Epoch: 9 [10000/80000 (12%)]	Loss: 0.695405
Train Epoch: 9 [20000/80000 (25%)]	Loss: 0.702356
Train Epoch: 9 [30000/80000 (38%)]	Loss: 0.702244
Train Epoch: 9 [40000/80000 (50%)]	Loss: 0.696443
Train Epoch: 9 [50000/80000 (62%)]	Loss: 0.704172
Train Epoch: 9 [60000/80000 (75%)]	Loss: 0.705754
Train Epoch: 9 [70000/80000 (88%)]	Loss: 0.700879

Test set: Average loss: 0.6997, Accuracy: 8536/20000 (42.680%)

Train Epoch: 10 [0/80000 (0%)]	Loss: 0.698163
Train Epoch: 10 [10000/80000 (12%)]	Loss: 0.702479
Train Epoch: 10 [20000/80000 (25%)]	Loss: 0.697485
Train Epoch: 10 [30000/80000 (38%)]	Loss: 0.702398
Train Epoch: 10 [40000/80000 (50%)]	Loss: 0.711448
Train Epoch: 10 [50000/80000 (62%)]	Loss: 0.702527
Train Epoch: 10 [60000/80000 (75%)]	Loss: 0.711168
Train Epoch: 10 [70000/80000 (88%)]	Loss: 0.697541

Test set: Average loss: 0.6996, Accuracy: 8548/20000 (42.740%)


 training DNN with 100000 data points and SGD lr=0.000100. 

Train Epoch: 1 [0/80000 (0%)]	Loss: 0.715197
Train Epoch: 1 [10000/80000 (12%)]	Loss: 0.711921
Train Epoch: 1 [20000/80000 (25%)]	Loss: 0.729711
Train Epoch: 1 [30000/80000 (38%)]	Loss: 0.724807
Train Epoch: 1 [40000/80000 (50%)]	Loss: 0.727388
Train Epoch: 1 [50000/80000 (62%)]	Loss: 0.724497
Train Epoch: 1 [60000/80000 (75%)]	Loss: 0.725045
Train Epoch: 1 [70000/80000 (88%)]	Loss: 0.727475

Test set: Average loss: 0.7087, Accuracy: 8248/20000 (41.240%)

Train Epoch: 2 [0/80000 (0%)]	Loss: 0.714319
Train Epoch: 2 [10000/80000 (12%)]	Loss: 0.731445
Train Epoch: 2 [20000/80000 (25%)]	Loss: 0.707774
Train Epoch: 2 [30000/80000 (38%)]	Loss: 0.714780
Train Epoch: 2 [40000/80000 (50%)]	Loss: 0.718108
Train Epoch: 2 [50000/80000 (62%)]	Loss: 0.720248
Train Epoch: 2 [60000/80000 (75%)]	Loss: 0.714495
Train Epoch: 2 [70000/80000 (88%)]	Loss: 0.715954

Test set: Average loss: 0.7062, Accuracy: 8342/20000 (41.710%)

Train Epoch: 3 [0/80000 (0%)]	Loss: 0.717270
Train Epoch: 3 [10000/80000 (12%)]	Loss: 0.719187
Train Epoch: 3 [20000/80000 (25%)]	Loss: 0.706191
Train Epoch: 3 [30000/80000 (38%)]	Loss: 0.716768
Train Epoch: 3 [40000/80000 (50%)]	Loss: 0.716282
Train Epoch: 3 [50000/80000 (62%)]	Loss: 0.714398
Train Epoch: 3 [60000/80000 (75%)]	Loss: 0.716676
Train Epoch: 3 [70000/80000 (88%)]	Loss: 0.716338

Test set: Average loss: 0.7040, Accuracy: 8496/20000 (42.480%)

Train Epoch: 4 [0/80000 (0%)]	Loss: 0.708547
Train Epoch: 4 [10000/80000 (12%)]	Loss: 0.712602
Train Epoch: 4 [20000/80000 (25%)]	Loss: 0.717987
Train Epoch: 4 [30000/80000 (38%)]	Loss: 0.714138
Train Epoch: 4 [40000/80000 (50%)]	Loss: 0.714872
Train Epoch: 4 [50000/80000 (62%)]	Loss: 0.707068
Train Epoch: 4 [60000/80000 (75%)]	Loss: 0.709471
Train Epoch: 4 [70000/80000 (88%)]	Loss: 0.719670

Test set: Average loss: 0.7020, Accuracy: 8648/20000 (43.240%)

Train Epoch: 5 [0/80000 (0%)]	Loss: 0.712123
Train Epoch: 5 [10000/80000 (12%)]	Loss: 0.705291
Train Epoch: 5 [20000/80000 (25%)]	Loss: 0.709001
Train Epoch: 5 [30000/80000 (38%)]	Loss: 0.702263
Train Epoch: 5 [40000/80000 (50%)]	Loss: 0.706440
Train Epoch: 5 [50000/80000 (62%)]	Loss: 0.704011
Train Epoch: 5 [60000/80000 (75%)]	Loss: 0.713797
Train Epoch: 5 [70000/80000 (88%)]	Loss: 0.713780

Test set: Average loss: 0.7002, Accuracy: 8851/20000 (44.255%)

Train Epoch: 6 [0/80000 (0%)]	Loss: 0.708955
Train Epoch: 6 [10000/80000 (12%)]	Loss: 0.701124
Train Epoch: 6 [20000/80000 (25%)]	Loss: 0.706962
Train Epoch: 6 [30000/80000 (38%)]	Loss: 0.703147
Train Epoch: 6 [40000/80000 (50%)]	Loss: 0.713594
Train Epoch: 6 [50000/80000 (62%)]	Loss: 0.714497
Train Epoch: 6 [60000/80000 (75%)]	Loss: 0.703184
Train Epoch: 6 [70000/80000 (88%)]	Loss: 0.703132

Test set: Average loss: 0.6985, Accuracy: 9027/20000 (45.135%)

Train Epoch: 7 [0/80000 (0%)]	Loss: 0.710458
Train Epoch: 7 [10000/80000 (12%)]	Loss: 0.706783
Train Epoch: 7 [20000/80000 (25%)]	Loss: 0.704416
Train Epoch: 7 [30000/80000 (38%)]	Loss: 0.708493
Train Epoch: 7 [40000/80000 (50%)]	Loss: 0.716165
Train Epoch: 7 [50000/80000 (62%)]	Loss: 0.706213
Train Epoch: 7 [60000/80000 (75%)]	Loss: 0.700028
Train Epoch: 7 [70000/80000 (88%)]	Loss: 0.713193

Test set: Average loss: 0.6969, Accuracy: 9210/20000 (46.050%)

Train Epoch: 8 [0/80000 (0%)]	Loss: 0.708336
Train Epoch: 8 [10000/80000 (12%)]	Loss: 0.705779
Train Epoch: 8 [20000/80000 (25%)]	Loss: 0.708531
Train Epoch: 8 [30000/80000 (38%)]	Loss: 0.707671
Train Epoch: 8 [40000/80000 (50%)]	Loss: 0.712592
Train Epoch: 8 [50000/80000 (62%)]	Loss: 0.710731
Train Epoch: 8 [60000/80000 (75%)]	Loss: 0.708170
Train Epoch: 8 [70000/80000 (88%)]	Loss: 0.701300

Test set: Average loss: 0.6954, Accuracy: 9392/20000 (46.960%)

Train Epoch: 9 [0/80000 (0%)]	Loss: 0.709463
Train Epoch: 9 [10000/80000 (12%)]	Loss: 0.704660
Train Epoch: 9 [20000/80000 (25%)]	Loss: 0.689310
Train Epoch: 9 [30000/80000 (38%)]	Loss: 0.703454
Train Epoch: 9 [40000/80000 (50%)]	Loss: 0.706950
Train Epoch: 9 [50000/80000 (62%)]	Loss: 0.701706
Train Epoch: 9 [60000/80000 (75%)]	Loss: 0.701360
Train Epoch: 9 [70000/80000 (88%)]	Loss: 0.700901

Test set: Average loss: 0.6939, Accuracy: 9581/20000 (47.905%)

Train Epoch: 10 [0/80000 (0%)]	Loss: 0.709202
Train Epoch: 10 [10000/80000 (12%)]	Loss: 0.709092
Train Epoch: 10 [20000/80000 (25%)]	Loss: 0.702193
Train Epoch: 10 [30000/80000 (38%)]	Loss: 0.702133
Train Epoch: 10 [40000/80000 (50%)]	Loss: 0.701327
Train Epoch: 10 [50000/80000 (62%)]	Loss: 0.705861
Train Epoch: 10 [60000/80000 (75%)]	Loss: 0.698883
Train Epoch: 10 [70000/80000 (88%)]	Loss: 0.700197

Test set: Average loss: 0.6924, Accuracy: 9760/20000 (48.800%)


 training DNN with 100000 data points and SGD lr=0.001000. 

Train Epoch: 1 [0/80000 (0%)]	Loss: 0.710853
Train Epoch: 1 [10000/80000 (12%)]	Loss: 0.703988
Train Epoch: 1 [20000/80000 (25%)]	Loss: 0.707492
Train Epoch: 1 [30000/80000 (38%)]	Loss: 0.699851
Train Epoch: 1 [40000/80000 (50%)]	Loss: 0.703508
Train Epoch: 1 [50000/80000 (62%)]	Loss: 0.701523
Train Epoch: 1 [60000/80000 (75%)]	Loss: 0.703516
Train Epoch: 1 [70000/80000 (88%)]	Loss: 0.686840

Test set: Average loss: 0.6848, Accuracy: 11189/20000 (55.945%)

Train Epoch: 2 [0/80000 (0%)]	Loss: 0.693794
Train Epoch: 2 [10000/80000 (12%)]	Loss: 0.690592
Train Epoch: 2 [20000/80000 (25%)]	Loss: 0.689504
Train Epoch: 2 [30000/80000 (38%)]	Loss: 0.687438
Train Epoch: 2 [40000/80000 (50%)]	Loss: 0.690289
Train Epoch: 2 [50000/80000 (62%)]	Loss: 0.688909
Train Epoch: 2 [60000/80000 (75%)]	Loss: 0.684075
Train Epoch: 2 [70000/80000 (88%)]	Loss: 0.691422

Test set: Average loss: 0.6729, Accuracy: 13286/20000 (66.430%)

Train Epoch: 3 [0/80000 (0%)]	Loss: 0.690993
Train Epoch: 3 [10000/80000 (12%)]	Loss: 0.689412
Train Epoch: 3 [20000/80000 (25%)]	Loss: 0.680717
Train Epoch: 3 [30000/80000 (38%)]	Loss: 0.681917
Train Epoch: 3 [40000/80000 (50%)]	Loss: 0.674909
Train Epoch: 3 [50000/80000 (62%)]	Loss: 0.683230
Train Epoch: 3 [60000/80000 (75%)]	Loss: 0.665544
Train Epoch: 3 [70000/80000 (88%)]	Loss: 0.685835

Test set: Average loss: 0.6628, Accuracy: 14070/20000 (70.350%)

Train Epoch: 4 [0/80000 (0%)]	Loss: 0.677483
Train Epoch: 4 [10000/80000 (12%)]	Loss: 0.681490
Train Epoch: 4 [20000/80000 (25%)]	Loss: 0.677488
Train Epoch: 4 [30000/80000 (38%)]	Loss: 0.667637
Train Epoch: 4 [40000/80000 (50%)]	Loss: 0.669382
Train Epoch: 4 [50000/80000 (62%)]	Loss: 0.670441
Train Epoch: 4 [60000/80000 (75%)]	Loss: 0.664033
Train Epoch: 4 [70000/80000 (88%)]	Loss: 0.669917

Test set: Average loss: 0.6535, Accuracy: 14414/20000 (72.070%)

Train Epoch: 5 [0/80000 (0%)]	Loss: 0.662508
Train Epoch: 5 [10000/80000 (12%)]	Loss: 0.663022
Train Epoch: 5 [20000/80000 (25%)]	Loss: 0.665074
Train Epoch: 5 [30000/80000 (38%)]	Loss: 0.660559
Train Epoch: 5 [40000/80000 (50%)]	Loss: 0.666832
Train Epoch: 5 [50000/80000 (62%)]	Loss: 0.664970
Train Epoch: 5 [60000/80000 (75%)]	Loss: 0.657169
Train Epoch: 5 [70000/80000 (88%)]	Loss: 0.667754

Test set: Average loss: 0.6448, Accuracy: 14529/20000 (72.645%)

Train Epoch: 6 [0/80000 (0%)]	Loss: 0.662098
Train Epoch: 6 [10000/80000 (12%)]	Loss: 0.661854
Train Epoch: 6 [20000/80000 (25%)]	Loss: 0.658346
Train Epoch: 6 [30000/80000 (38%)]	Loss: 0.659743
Train Epoch: 6 [40000/80000 (50%)]	Loss: 0.659351
Train Epoch: 6 [50000/80000 (62%)]	Loss: 0.651273
Train Epoch: 6 [60000/80000 (75%)]	Loss: 0.648752
Train Epoch: 6 [70000/80000 (88%)]	Loss: 0.650942

Test set: Average loss: 0.6363, Accuracy: 14627/20000 (73.135%)

Train Epoch: 7 [0/80000 (0%)]	Loss: 0.650276
Train Epoch: 7 [10000/80000 (12%)]	Loss: 0.642630
Train Epoch: 7 [20000/80000 (25%)]	Loss: 0.647791
Train Epoch: 7 [30000/80000 (38%)]	Loss: 0.652298
Train Epoch: 7 [40000/80000 (50%)]	Loss: 0.638959
Train Epoch: 7 [50000/80000 (62%)]	Loss: 0.639897
Train Epoch: 7 [60000/80000 (75%)]	Loss: 0.638109
Train Epoch: 7 [70000/80000 (88%)]	Loss: 0.644963

Test set: Average loss: 0.6282, Accuracy: 14665/20000 (73.325%)

Train Epoch: 8 [0/80000 (0%)]	Loss: 0.651271
Train Epoch: 8 [10000/80000 (12%)]	Loss: 0.638376
Train Epoch: 8 [20000/80000 (25%)]	Loss: 0.645874
Train Epoch: 8 [30000/80000 (38%)]	Loss: 0.638896
Train Epoch: 8 [40000/80000 (50%)]	Loss: 0.637035
Train Epoch: 8 [50000/80000 (62%)]	Loss: 0.636409
Train Epoch: 8 [60000/80000 (75%)]	Loss: 0.636783
Train Epoch: 8 [70000/80000 (88%)]	Loss: 0.644974

Test set: Average loss: 0.6201, Accuracy: 14714/20000 (73.570%)

Train Epoch: 9 [0/80000 (0%)]	Loss: 0.645297
Train Epoch: 9 [10000/80000 (12%)]	Loss: 0.637906
Train Epoch: 9 [20000/80000 (25%)]	Loss: 0.640790
Train Epoch: 9 [30000/80000 (38%)]	Loss: 0.627469
Train Epoch: 9 [40000/80000 (50%)]	Loss: 0.638363
Train Epoch: 9 [50000/80000 (62%)]	Loss: 0.633748
Train Epoch: 9 [60000/80000 (75%)]	Loss: 0.643103
Train Epoch: 9 [70000/80000 (88%)]	Loss: 0.636656

Test set: Average loss: 0.6121, Accuracy: 14779/20000 (73.895%)

Train Epoch: 10 [0/80000 (0%)]	Loss: 0.633914
Train Epoch: 10 [10000/80000 (12%)]	Loss: 0.632493
Train Epoch: 10 [20000/80000 (25%)]	Loss: 0.627166
Train Epoch: 10 [30000/80000 (38%)]	Loss: 0.632092
Train Epoch: 10 [40000/80000 (50%)]	Loss: 0.636172
Train Epoch: 10 [50000/80000 (62%)]	Loss: 0.618695
Train Epoch: 10 [60000/80000 (75%)]	Loss: 0.630732
Train Epoch: 10 [70000/80000 (88%)]	Loss: 0.642953

Test set: Average loss: 0.6047, Accuracy: 14810/20000 (74.050%)


 training DNN with 100000 data points and SGD lr=0.010000. 

Train Epoch: 1 [0/80000 (0%)]	Loss: 0.697973
Train Epoch: 1 [10000/80000 (12%)]	Loss: 0.689287
Train Epoch: 1 [20000/80000 (25%)]	Loss: 0.678359
Train Epoch: 1 [30000/80000 (38%)]	Loss: 0.670037
Train Epoch: 1 [40000/80000 (50%)]	Loss: 0.662208
Train Epoch: 1 [50000/80000 (62%)]	Loss: 0.655228
Train Epoch: 1 [60000/80000 (75%)]	Loss: 0.648531
Train Epoch: 1 [70000/80000 (88%)]	Loss: 0.635769

Test set: Average loss: 0.6114, Accuracy: 14907/20000 (74.535%)

Train Epoch: 2 [0/80000 (0%)]	Loss: 0.636103
Train Epoch: 2 [10000/80000 (12%)]	Loss: 0.626280
Train Epoch: 2 [20000/80000 (25%)]	Loss: 0.612292
Train Epoch: 2 [30000/80000 (38%)]	Loss: 0.616145
Train Epoch: 2 [40000/80000 (50%)]	Loss: 0.607718
Train Epoch: 2 [50000/80000 (62%)]	Loss: 0.587785
Train Epoch: 2 [60000/80000 (75%)]	Loss: 0.613270
Train Epoch: 2 [70000/80000 (88%)]	Loss: 0.579698

Test set: Average loss: 0.5489, Accuracy: 15207/20000 (76.035%)

Train Epoch: 3 [0/80000 (0%)]	Loss: 0.588873
Train Epoch: 3 [10000/80000 (12%)]	Loss: 0.569671
Train Epoch: 3 [20000/80000 (25%)]	Loss: 0.578632
Train Epoch: 3 [30000/80000 (38%)]	Loss: 0.576780
Train Epoch: 3 [40000/80000 (50%)]	Loss: 0.582997
Train Epoch: 3 [50000/80000 (62%)]	Loss: 0.574457
Train Epoch: 3 [60000/80000 (75%)]	Loss: 0.543472
Train Epoch: 3 [70000/80000 (88%)]	Loss: 0.528735

Test set: Average loss: 0.5137, Accuracy: 15339/20000 (76.695%)

Train Epoch: 4 [0/80000 (0%)]	Loss: 0.551613
Train Epoch: 4 [10000/80000 (12%)]	Loss: 0.543068
Train Epoch: 4 [20000/80000 (25%)]	Loss: 0.548906
Train Epoch: 4 [30000/80000 (38%)]	Loss: 0.562617
Train Epoch: 4 [40000/80000 (50%)]	Loss: 0.515420
Train Epoch: 4 [50000/80000 (62%)]	Loss: 0.561311
Train Epoch: 4 [60000/80000 (75%)]	Loss: 0.559609
Train Epoch: 4 [70000/80000 (88%)]	Loss: 0.551798

Test set: Average loss: 0.4963, Accuracy: 15470/20000 (77.350%)

Train Epoch: 5 [0/80000 (0%)]	Loss: 0.549332
Train Epoch: 5 [10000/80000 (12%)]	Loss: 0.525346
Train Epoch: 5 [20000/80000 (25%)]	Loss: 0.552634
Train Epoch: 5 [30000/80000 (38%)]	Loss: 0.563371
Train Epoch: 5 [40000/80000 (50%)]	Loss: 0.564668
Train Epoch: 5 [50000/80000 (62%)]	Loss: 0.568728
Train Epoch: 5 [60000/80000 (75%)]	Loss: 0.560736
Train Epoch: 5 [70000/80000 (88%)]	Loss: 0.501028

Test set: Average loss: 0.4865, Accuracy: 15535/20000 (77.675%)

Train Epoch: 6 [0/80000 (0%)]	Loss: 0.564258
Train Epoch: 6 [10000/80000 (12%)]	Loss: 0.520596
Train Epoch: 6 [20000/80000 (25%)]	Loss: 0.547322
Train Epoch: 6 [30000/80000 (38%)]	Loss: 0.515506
Train Epoch: 6 [40000/80000 (50%)]	Loss: 0.550544
Train Epoch: 6 [50000/80000 (62%)]	Loss: 0.505403
Train Epoch: 6 [60000/80000 (75%)]	Loss: 0.534858
Train Epoch: 6 [70000/80000 (88%)]	Loss: 0.502775

Test set: Average loss: 0.4789, Accuracy: 15588/20000 (77.940%)

Train Epoch: 7 [0/80000 (0%)]	Loss: 0.530125
Train Epoch: 7 [10000/80000 (12%)]	Loss: 0.528042
Train Epoch: 7 [20000/80000 (25%)]	Loss: 0.523079
Train Epoch: 7 [30000/80000 (38%)]	Loss: 0.482682
Train Epoch: 7 [40000/80000 (50%)]	Loss: 0.503262
Train Epoch: 7 [50000/80000 (62%)]	Loss: 0.516694
Train Epoch: 7 [60000/80000 (75%)]	Loss: 0.533552
Train Epoch: 7 [70000/80000 (88%)]	Loss: 0.510181

Test set: Average loss: 0.4749, Accuracy: 15621/20000 (78.105%)

Train Epoch: 8 [0/80000 (0%)]	Loss: 0.506149
Train Epoch: 8 [10000/80000 (12%)]	Loss: 0.521474
Train Epoch: 8 [20000/80000 (25%)]	Loss: 0.491547
Train Epoch: 8 [30000/80000 (38%)]	Loss: 0.511403
Train Epoch: 8 [40000/80000 (50%)]	Loss: 0.500965
Train Epoch: 8 [50000/80000 (62%)]	Loss: 0.478549
Train Epoch: 8 [60000/80000 (75%)]	Loss: 0.516275
Train Epoch: 8 [70000/80000 (88%)]	Loss: 0.492698

Test set: Average loss: 0.4703, Accuracy: 15684/20000 (78.420%)

Train Epoch: 9 [0/80000 (0%)]	Loss: 0.531255
Train Epoch: 9 [10000/80000 (12%)]	Loss: 0.506924
Train Epoch: 9 [20000/80000 (25%)]	Loss: 0.502441
Train Epoch: 9 [30000/80000 (38%)]	Loss: 0.494101
Train Epoch: 9 [40000/80000 (50%)]	Loss: 0.483607
Train Epoch: 9 [50000/80000 (62%)]	Loss: 0.531680
Train Epoch: 9 [60000/80000 (75%)]	Loss: 0.490221
Train Epoch: 9 [70000/80000 (88%)]	Loss: 0.494850

Test set: Average loss: 0.4676, Accuracy: 15705/20000 (78.525%)

Train Epoch: 10 [0/80000 (0%)]	Loss: 0.505682
Train Epoch: 10 [10000/80000 (12%)]	Loss: 0.489378
Train Epoch: 10 [20000/80000 (25%)]	Loss: 0.496558
Train Epoch: 10 [30000/80000 (38%)]	Loss: 0.510254
Train Epoch: 10 [40000/80000 (50%)]	Loss: 0.496754
Train Epoch: 10 [50000/80000 (62%)]	Loss: 0.529930
Train Epoch: 10 [60000/80000 (75%)]	Loss: 0.502369
Train Epoch: 10 [70000/80000 (88%)]	Loss: 0.487612

Test set: Average loss: 0.4658, Accuracy: 15719/20000 (78.595%)


 training DNN with 100000 data points and SGD lr=0.100000. 

Train Epoch: 1 [0/80000 (0%)]	Loss: 0.705091
Train Epoch: 1 [10000/80000 (12%)]	Loss: 0.641505
Train Epoch: 1 [20000/80000 (25%)]	Loss: 0.580319
Train Epoch: 1 [30000/80000 (38%)]	Loss: 0.557262
Train Epoch: 1 [40000/80000 (50%)]	Loss: 0.549322
Train Epoch: 1 [50000/80000 (62%)]	Loss: 0.527447
Train Epoch: 1 [60000/80000 (75%)]	Loss: 0.502881
Train Epoch: 1 [70000/80000 (88%)]	Loss: 0.506342

Test set: Average loss: 0.4698, Accuracy: 15614/20000 (78.070%)

Train Epoch: 2 [0/80000 (0%)]	Loss: 0.493702
Train Epoch: 2 [10000/80000 (12%)]	Loss: 0.505257
Train Epoch: 2 [20000/80000 (25%)]	Loss: 0.481080
Train Epoch: 2 [30000/80000 (38%)]	Loss: 0.512082
Train Epoch: 2 [40000/80000 (50%)]	Loss: 0.471982
Train Epoch: 2 [50000/80000 (62%)]	Loss: 0.506896
Train Epoch: 2 [60000/80000 (75%)]	Loss: 0.497945
Train Epoch: 2 [70000/80000 (88%)]	Loss: 0.500737

Test set: Average loss: 0.4553, Accuracy: 15809/20000 (79.045%)

Train Epoch: 3 [0/80000 (0%)]	Loss: 0.487297
Train Epoch: 3 [10000/80000 (12%)]	Loss: 0.472463
Train Epoch: 3 [20000/80000 (25%)]	Loss: 0.457577
Train Epoch: 3 [30000/80000 (38%)]	Loss: 0.479991
Train Epoch: 3 [40000/80000 (50%)]	Loss: 0.494305
Train Epoch: 3 [50000/80000 (62%)]	Loss: 0.464404
Train Epoch: 3 [60000/80000 (75%)]	Loss: 0.479279
Train Epoch: 3 [70000/80000 (88%)]	Loss: 0.459238

Test set: Average loss: 0.4519, Accuracy: 15826/20000 (79.130%)

Train Epoch: 4 [0/80000 (0%)]	Loss: 0.466627
Train Epoch: 4 [10000/80000 (12%)]	Loss: 0.471862
Train Epoch: 4 [20000/80000 (25%)]	Loss: 0.451782
Train Epoch: 4 [30000/80000 (38%)]	Loss: 0.476699
Train Epoch: 4 [40000/80000 (50%)]	Loss: 0.480063
Train Epoch: 4 [50000/80000 (62%)]	Loss: 0.461674
Train Epoch: 4 [60000/80000 (75%)]	Loss: 0.474348
Train Epoch: 4 [70000/80000 (88%)]	Loss: 0.451285

Test set: Average loss: 0.4491, Accuracy: 15872/20000 (79.360%)

Train Epoch: 5 [0/80000 (0%)]	Loss: 0.454546
Train Epoch: 5 [10000/80000 (12%)]	Loss: 0.431953
Train Epoch: 5 [20000/80000 (25%)]	Loss: 0.423571
Train Epoch: 5 [30000/80000 (38%)]	Loss: 0.458014
Train Epoch: 5 [40000/80000 (50%)]	Loss: 0.449383
Train Epoch: 5 [50000/80000 (62%)]	Loss: 0.441713
Train Epoch: 5 [60000/80000 (75%)]	Loss: 0.455981
Train Epoch: 5 [70000/80000 (88%)]	Loss: 0.445281

Test set: Average loss: 0.4477, Accuracy: 15878/20000 (79.390%)

Train Epoch: 6 [0/80000 (0%)]	Loss: 0.476829
Train Epoch: 6 [10000/80000 (12%)]	Loss: 0.440844
Train Epoch: 6 [20000/80000 (25%)]	Loss: 0.450948
Train Epoch: 6 [30000/80000 (38%)]	Loss: 0.464564
Train Epoch: 6 [40000/80000 (50%)]	Loss: 0.452788
Train Epoch: 6 [50000/80000 (62%)]	Loss: 0.458984
Train Epoch: 6 [60000/80000 (75%)]	Loss: 0.443346
Train Epoch: 6 [70000/80000 (88%)]	Loss: 0.461984

Test set: Average loss: 0.4460, Accuracy: 15895/20000 (79.475%)

Train Epoch: 7 [0/80000 (0%)]	Loss: 0.458578
Train Epoch: 7 [10000/80000 (12%)]	Loss: 0.436979
Train Epoch: 7 [20000/80000 (25%)]	Loss: 0.458328
Train Epoch: 7 [30000/80000 (38%)]	Loss: 0.457037
Train Epoch: 7 [40000/80000 (50%)]	Loss: 0.483248
Train Epoch: 7 [50000/80000 (62%)]	Loss: 0.462219
Train Epoch: 7 [60000/80000 (75%)]	Loss: 0.442516
Train Epoch: 7 [70000/80000 (88%)]	Loss: 0.446028

Test set: Average loss: 0.4449, Accuracy: 15916/20000 (79.580%)

Train Epoch: 8 [0/80000 (0%)]	Loss: 0.451255
Train Epoch: 8 [10000/80000 (12%)]	Loss: 0.433276
Train Epoch: 8 [20000/80000 (25%)]	Loss: 0.485846
Train Epoch: 8 [30000/80000 (38%)]	Loss: 0.445357
Train Epoch: 8 [40000/80000 (50%)]	Loss: 0.429605
Train Epoch: 8 [50000/80000 (62%)]	Loss: 0.443655
Train Epoch: 8 [60000/80000 (75%)]	Loss: 0.436730
Train Epoch: 8 [70000/80000 (88%)]	Loss: 0.469937

Test set: Average loss: 0.4445, Accuracy: 15914/20000 (79.570%)

Train Epoch: 9 [0/80000 (0%)]	Loss: 0.463443
Train Epoch: 9 [10000/80000 (12%)]	Loss: 0.460738
Train Epoch: 9 [20000/80000 (25%)]	Loss: 0.430145
Train Epoch: 9 [30000/80000 (38%)]	Loss: 0.451020
Train Epoch: 9 [40000/80000 (50%)]	Loss: 0.462381
Train Epoch: 9 [50000/80000 (62%)]	Loss: 0.466396
Train Epoch: 9 [60000/80000 (75%)]	Loss: 0.448938
Train Epoch: 9 [70000/80000 (88%)]	Loss: 0.468543

Test set: Average loss: 0.4433, Accuracy: 15933/20000 (79.665%)

Train Epoch: 10 [0/80000 (0%)]	Loss: 0.433695
Train Epoch: 10 [10000/80000 (12%)]	Loss: 0.487441
Train Epoch: 10 [20000/80000 (25%)]	Loss: 0.454066
Train Epoch: 10 [30000/80000 (38%)]	Loss: 0.450409
Train Epoch: 10 [40000/80000 (50%)]	Loss: 0.455621
Train Epoch: 10 [50000/80000 (62%)]	Loss: 0.426238
Train Epoch: 10 [60000/80000 (75%)]	Loss: 0.462113
Train Epoch: 10 [70000/80000 (88%)]	Loss: 0.440993

Test set: Average loss: 0.4436, Accuracy: 15917/20000 (79.585%)

Training on 160000 examples
Using both high and low level features
Testing on 40000 examples
Using both high and low level features

 training DNN with 200000 data points and SGD lr=0.000010. 

Train Epoch: 1 [0/160000 (0%)]	Loss: 0.697536
Train Epoch: 1 [20000/160000 (12%)]	Loss: 0.694769
Train Epoch: 1 [40000/160000 (25%)]	Loss: 0.695942
Train Epoch: 1 [60000/160000 (38%)]	Loss: 0.695974
Train Epoch: 1 [80000/160000 (50%)]	Loss: 0.687821
Train Epoch: 1 [100000/160000 (62%)]	Loss: 0.689334
Train Epoch: 1 [120000/160000 (75%)]	Loss: 0.695128
Train Epoch: 1 [140000/160000 (88%)]	Loss: 0.693733

Test set: Average loss: 0.6855, Accuracy: 22418/40000 (56.045%)

Train Epoch: 2 [0/160000 (0%)]	Loss: 0.690398
Train Epoch: 2 [20000/160000 (12%)]	Loss: 0.694173
Train Epoch: 2 [40000/160000 (25%)]	Loss: 0.692116
Train Epoch: 2 [60000/160000 (38%)]	Loss: 0.691964
Train Epoch: 2 [80000/160000 (50%)]	Loss: 0.691619
Train Epoch: 2 [100000/160000 (62%)]	Loss: 0.690516
Train Epoch: 2 [120000/160000 (75%)]	Loss: 0.693235
Train Epoch: 2 [140000/160000 (88%)]	Loss: 0.694582

Test set: Average loss: 0.6854, Accuracy: 22448/40000 (56.120%)

Train Epoch: 3 [0/160000 (0%)]	Loss: 0.696227
Train Epoch: 3 [20000/160000 (12%)]	Loss: 0.694529
Train Epoch: 3 [40000/160000 (25%)]	Loss: 0.690709
Train Epoch: 3 [60000/160000 (38%)]	Loss: 0.689253
Train Epoch: 3 [80000/160000 (50%)]	Loss: 0.690921
Train Epoch: 3 [100000/160000 (62%)]	Loss: 0.685167
Train Epoch: 3 [120000/160000 (75%)]	Loss: 0.692460
Train Epoch: 3 [140000/160000 (88%)]	Loss: 0.686860

Test set: Average loss: 0.6853, Accuracy: 22474/40000 (56.185%)

Train Epoch: 4 [0/160000 (0%)]	Loss: 0.688780
Train Epoch: 4 [20000/160000 (12%)]	Loss: 0.693148
Train Epoch: 4 [40000/160000 (25%)]	Loss: 0.692194
Train Epoch: 4 [60000/160000 (38%)]	Loss: 0.692709
Train Epoch: 4 [80000/160000 (50%)]	Loss: 0.693655
Train Epoch: 4 [100000/160000 (62%)]	Loss: 0.690548
Train Epoch: 4 [120000/160000 (75%)]	Loss: 0.690444
Train Epoch: 4 [140000/160000 (88%)]	Loss: 0.689332

Test set: Average loss: 0.6851, Accuracy: 22497/40000 (56.242%)

Train Epoch: 5 [0/160000 (0%)]	Loss: 0.688800
Train Epoch: 5 [20000/160000 (12%)]	Loss: 0.692959
Train Epoch: 5 [40000/160000 (25%)]	Loss: 0.690799
Train Epoch: 5 [60000/160000 (38%)]	Loss: 0.691604
Train Epoch: 5 [80000/160000 (50%)]	Loss: 0.693255
Train Epoch: 5 [100000/160000 (62%)]	Loss: 0.687228
Train Epoch: 5 [120000/160000 (75%)]	Loss: 0.689395
Train Epoch: 5 [140000/160000 (88%)]	Loss: 0.695083

Test set: Average loss: 0.6850, Accuracy: 22529/40000 (56.322%)

Train Epoch: 6 [0/160000 (0%)]	Loss: 0.689608
Train Epoch: 6 [20000/160000 (12%)]	Loss: 0.685819
Train Epoch: 6 [40000/160000 (25%)]	Loss: 0.688747
Train Epoch: 6 [60000/160000 (38%)]	Loss: 0.692951
Train Epoch: 6 [80000/160000 (50%)]	Loss: 0.695513
Train Epoch: 6 [100000/160000 (62%)]	Loss: 0.698390
Train Epoch: 6 [120000/160000 (75%)]	Loss: 0.697410
Train Epoch: 6 [140000/160000 (88%)]	Loss: 0.694441

Test set: Average loss: 0.6849, Accuracy: 22563/40000 (56.407%)

Train Epoch: 7 [0/160000 (0%)]	Loss: 0.695451
Train Epoch: 7 [20000/160000 (12%)]	Loss: 0.690968
Train Epoch: 7 [40000/160000 (25%)]	Loss: 0.694902
Train Epoch: 7 [60000/160000 (38%)]	Loss: 0.696121
Train Epoch: 7 [80000/160000 (50%)]	Loss: 0.692092
Train Epoch: 7 [100000/160000 (62%)]	Loss: 0.690247
Train Epoch: 7 [120000/160000 (75%)]	Loss: 0.695067
Train Epoch: 7 [140000/160000 (88%)]	Loss: 0.691260

Test set: Average loss: 0.6848, Accuracy: 22594/40000 (56.485%)

Train Epoch: 8 [0/160000 (0%)]	Loss: 0.691153
Train Epoch: 8 [20000/160000 (12%)]	Loss: 0.691588
Train Epoch: 8 [40000/160000 (25%)]	Loss: 0.692792
Train Epoch: 8 [60000/160000 (38%)]	Loss: 0.695482
Train Epoch: 8 [80000/160000 (50%)]	Loss: 0.694909
Train Epoch: 8 [100000/160000 (62%)]	Loss: 0.694001
Train Epoch: 8 [120000/160000 (75%)]	Loss: 0.691612
Train Epoch: 8 [140000/160000 (88%)]	Loss: 0.689127

Test set: Average loss: 0.6846, Accuracy: 22617/40000 (56.542%)

Train Epoch: 9 [0/160000 (0%)]	Loss: 0.699426
Train Epoch: 9 [20000/160000 (12%)]	Loss: 0.686623
Train Epoch: 9 [40000/160000 (25%)]	Loss: 0.695512
Train Epoch: 9 [60000/160000 (38%)]	Loss: 0.692502
Train Epoch: 9 [80000/160000 (50%)]	Loss: 0.686093
Train Epoch: 9 [100000/160000 (62%)]	Loss: 0.693720
Train Epoch: 9 [120000/160000 (75%)]	Loss: 0.691294
Train Epoch: 9 [140000/160000 (88%)]	Loss: 0.691889

Test set: Average loss: 0.6845, Accuracy: 22653/40000 (56.633%)

Train Epoch: 10 [0/160000 (0%)]	Loss: 0.692374
Train Epoch: 10 [20000/160000 (12%)]	Loss: 0.693439
Train Epoch: 10 [40000/160000 (25%)]	Loss: 0.694436
Train Epoch: 10 [60000/160000 (38%)]	Loss: 0.687419
Train Epoch: 10 [80000/160000 (50%)]	Loss: 0.692452
Train Epoch: 10 [100000/160000 (62%)]	Loss: 0.687066
Train Epoch: 10 [120000/160000 (75%)]	Loss: 0.691027
Train Epoch: 10 [140000/160000 (88%)]	Loss: 0.688050

Test set: Average loss: 0.6844, Accuracy: 22670/40000 (56.675%)


 training DNN with 200000 data points and SGD lr=0.000100. 

Train Epoch: 1 [0/160000 (0%)]	Loss: 0.697540
Train Epoch: 1 [20000/160000 (12%)]	Loss: 0.697083
Train Epoch: 1 [40000/160000 (25%)]	Loss: 0.695987
Train Epoch: 1 [60000/160000 (38%)]	Loss: 0.690630
Train Epoch: 1 [80000/160000 (50%)]	Loss: 0.695427
Train Epoch: 1 [100000/160000 (62%)]	Loss: 0.693449
Train Epoch: 1 [120000/160000 (75%)]	Loss: 0.691263
Train Epoch: 1 [140000/160000 (88%)]	Loss: 0.691844

Test set: Average loss: 0.6901, Accuracy: 21173/40000 (52.932%)

Train Epoch: 2 [0/160000 (0%)]	Loss: 0.692849
Train Epoch: 2 [20000/160000 (12%)]	Loss: 0.695949
Train Epoch: 2 [40000/160000 (25%)]	Loss: 0.693565
Train Epoch: 2 [60000/160000 (38%)]	Loss: 0.690521
Train Epoch: 2 [80000/160000 (50%)]	Loss: 0.692377
Train Epoch: 2 [100000/160000 (62%)]	Loss: 0.694584
Train Epoch: 2 [120000/160000 (75%)]	Loss: 0.692072
Train Epoch: 2 [140000/160000 (88%)]	Loss: 0.691132

Test set: Average loss: 0.6891, Accuracy: 21462/40000 (53.655%)

Train Epoch: 3 [0/160000 (0%)]	Loss: 0.695255
Train Epoch: 3 [20000/160000 (12%)]	Loss: 0.695031
Train Epoch: 3 [40000/160000 (25%)]	Loss: 0.690024
Train Epoch: 3 [60000/160000 (38%)]	Loss: 0.693541
Train Epoch: 3 [80000/160000 (50%)]	Loss: 0.691810
Train Epoch: 3 [100000/160000 (62%)]	Loss: 0.694084
Train Epoch: 3 [120000/160000 (75%)]	Loss: 0.697059
Train Epoch: 3 [140000/160000 (88%)]	Loss: 0.692586

Test set: Average loss: 0.6880, Accuracy: 21852/40000 (54.630%)

Train Epoch: 4 [0/160000 (0%)]	Loss: 0.696967
Train Epoch: 4 [20000/160000 (12%)]	Loss: 0.692188
Train Epoch: 4 [40000/160000 (25%)]	Loss: 0.685209
Train Epoch: 4 [60000/160000 (38%)]	Loss: 0.690242
Train Epoch: 4 [80000/160000 (50%)]	Loss: 0.695247
Train Epoch: 4 [100000/160000 (62%)]	Loss: 0.697769
Train Epoch: 4 [120000/160000 (75%)]	Loss: 0.690691
Train Epoch: 4 [140000/160000 (88%)]	Loss: 0.691298

Test set: Average loss: 0.6870, Accuracy: 22193/40000 (55.483%)

Train Epoch: 5 [0/160000 (0%)]	Loss: 0.689351
Train Epoch: 5 [20000/160000 (12%)]	Loss: 0.691943
Train Epoch: 5 [40000/160000 (25%)]	Loss: 0.695891
Train Epoch: 5 [60000/160000 (38%)]	Loss: 0.689337
Train Epoch: 5 [80000/160000 (50%)]	Loss: 0.689776
Train Epoch: 5 [100000/160000 (62%)]	Loss: 0.689858
Train Epoch: 5 [120000/160000 (75%)]	Loss: 0.687728
Train Epoch: 5 [140000/160000 (88%)]	Loss: 0.685325

Test set: Average loss: 0.6860, Accuracy: 22510/40000 (56.275%)

Train Epoch: 6 [0/160000 (0%)]	Loss: 0.689849
Train Epoch: 6 [20000/160000 (12%)]	Loss: 0.690648
Train Epoch: 6 [40000/160000 (25%)]	Loss: 0.689729
Train Epoch: 6 [60000/160000 (38%)]	Loss: 0.694689
Train Epoch: 6 [80000/160000 (50%)]	Loss: 0.686267
Train Epoch: 6 [100000/160000 (62%)]	Loss: 0.691425
Train Epoch: 6 [120000/160000 (75%)]	Loss: 0.687313
Train Epoch: 6 [140000/160000 (88%)]	Loss: 0.687692

Test set: Average loss: 0.6851, Accuracy: 22815/40000 (57.038%)

Train Epoch: 7 [0/160000 (0%)]	Loss: 0.689807
Train Epoch: 7 [20000/160000 (12%)]	Loss: 0.688712
Train Epoch: 7 [40000/160000 (25%)]	Loss: 0.689868
Train Epoch: 7 [60000/160000 (38%)]	Loss: 0.686265
Train Epoch: 7 [80000/160000 (50%)]	Loss: 0.687202
Train Epoch: 7 [100000/160000 (62%)]	Loss: 0.687460
Train Epoch: 7 [120000/160000 (75%)]	Loss: 0.689398
Train Epoch: 7 [140000/160000 (88%)]	Loss: 0.692314

Test set: Average loss: 0.6841, Accuracy: 23103/40000 (57.758%)

Train Epoch: 8 [0/160000 (0%)]	Loss: 0.686476
Train Epoch: 8 [20000/160000 (12%)]	Loss: 0.687612
Train Epoch: 8 [40000/160000 (25%)]	Loss: 0.685242
Train Epoch: 8 [60000/160000 (38%)]	Loss: 0.688446
Train Epoch: 8 [80000/160000 (50%)]	Loss: 0.689180
Train Epoch: 8 [100000/160000 (62%)]	Loss: 0.681610
Train Epoch: 8 [120000/160000 (75%)]	Loss: 0.684914
Train Epoch: 8 [140000/160000 (88%)]	Loss: 0.686573

Test set: Average loss: 0.6832, Accuracy: 23361/40000 (58.403%)

Train Epoch: 9 [0/160000 (0%)]	Loss: 0.688832
Train Epoch: 9 [20000/160000 (12%)]	Loss: 0.686388
Train Epoch: 9 [40000/160000 (25%)]	Loss: 0.685628
Train Epoch: 9 [60000/160000 (38%)]	Loss: 0.688939
Train Epoch: 9 [80000/160000 (50%)]	Loss: 0.689757
Train Epoch: 9 [100000/160000 (62%)]	Loss: 0.687121
Train Epoch: 9 [120000/160000 (75%)]	Loss: 0.688064
Train Epoch: 9 [140000/160000 (88%)]	Loss: 0.688722

Test set: Average loss: 0.6822, Accuracy: 23686/40000 (59.215%)

Train Epoch: 10 [0/160000 (0%)]	Loss: 0.688352
Train Epoch: 10 [20000/160000 (12%)]	Loss: 0.687095
Train Epoch: 10 [40000/160000 (25%)]	Loss: 0.688700
Train Epoch: 10 [60000/160000 (38%)]	Loss: 0.689241
Train Epoch: 10 [80000/160000 (50%)]	Loss: 0.685562
Train Epoch: 10 [100000/160000 (62%)]	Loss: 0.692156
Train Epoch: 10 [120000/160000 (75%)]	Loss: 0.686256
Train Epoch: 10 [140000/160000 (88%)]	Loss: 0.684139

Test set: Average loss: 0.6813, Accuracy: 23973/40000 (59.932%)


 training DNN with 200000 data points and SGD lr=0.001000. 

Train Epoch: 1 [0/160000 (0%)]	Loss: 0.711210
Train Epoch: 1 [20000/160000 (12%)]	Loss: 0.708529
Train Epoch: 1 [40000/160000 (25%)]	Loss: 0.709962
Train Epoch: 1 [60000/160000 (38%)]	Loss: 0.709772
Train Epoch: 1 [80000/160000 (50%)]	Loss: 0.709394
Train Epoch: 1 [100000/160000 (62%)]	Loss: 0.701491
Train Epoch: 1 [120000/160000 (75%)]	Loss: 0.702403
Train Epoch: 1 [140000/160000 (88%)]	Loss: 0.698894

Test set: Average loss: 0.6927, Accuracy: 19903/40000 (49.758%)

Train Epoch: 2 [0/160000 (0%)]	Loss: 0.695125
Train Epoch: 2 [20000/160000 (12%)]	Loss: 0.699186
Train Epoch: 2 [40000/160000 (25%)]	Loss: 0.701044
Train Epoch: 2 [60000/160000 (38%)]	Loss: 0.694566
Train Epoch: 2 [80000/160000 (50%)]	Loss: 0.694051
Train Epoch: 2 [100000/160000 (62%)]	Loss: 0.691268
Train Epoch: 2 [120000/160000 (75%)]	Loss: 0.693306
Train Epoch: 2 [140000/160000 (88%)]	Loss: 0.689296

Test set: Average loss: 0.6836, Accuracy: 24090/40000 (60.225%)

Train Epoch: 3 [0/160000 (0%)]	Loss: 0.685483
Train Epoch: 3 [20000/160000 (12%)]	Loss: 0.689562
Train Epoch: 3 [40000/160000 (25%)]	Loss: 0.693319
Train Epoch: 3 [60000/160000 (38%)]	Loss: 0.685696
Train Epoch: 3 [80000/160000 (50%)]	Loss: 0.689653
Train Epoch: 3 [100000/160000 (62%)]	Loss: 0.691741
Train Epoch: 3 [120000/160000 (75%)]	Loss: 0.687653
Train Epoch: 3 [140000/160000 (88%)]	Loss: 0.686184

Test set: Average loss: 0.6754, Accuracy: 27155/40000 (67.888%)

Train Epoch: 4 [0/160000 (0%)]	Loss: 0.679059
Train Epoch: 4 [20000/160000 (12%)]	Loss: 0.682266
Train Epoch: 4 [40000/160000 (25%)]	Loss: 0.678426
Train Epoch: 4 [60000/160000 (38%)]	Loss: 0.679051
Train Epoch: 4 [80000/160000 (50%)]	Loss: 0.681175
Train Epoch: 4 [100000/160000 (62%)]	Loss: 0.678167
Train Epoch: 4 [120000/160000 (75%)]	Loss: 0.677122
Train Epoch: 4 [140000/160000 (88%)]	Loss: 0.674250

Test set: Average loss: 0.6677, Accuracy: 28763/40000 (71.907%)

Train Epoch: 5 [0/160000 (0%)]	Loss: 0.678272
Train Epoch: 5 [20000/160000 (12%)]	Loss: 0.680418
Train Epoch: 5 [40000/160000 (25%)]	Loss: 0.682454
Train Epoch: 5 [60000/160000 (38%)]	Loss: 0.679521
Train Epoch: 5 [80000/160000 (50%)]	Loss: 0.671055
Train Epoch: 5 [100000/160000 (62%)]	Loss: 0.672802
Train Epoch: 5 [120000/160000 (75%)]	Loss: 0.669698
Train Epoch: 5 [140000/160000 (88%)]	Loss: 0.671952

Test set: Average loss: 0.6602, Accuracy: 29485/40000 (73.713%)

Train Epoch: 6 [0/160000 (0%)]	Loss: 0.669765
Train Epoch: 6 [20000/160000 (12%)]	Loss: 0.665902
Train Epoch: 6 [40000/160000 (25%)]	Loss: 0.670903
Train Epoch: 6 [60000/160000 (38%)]	Loss: 0.669754
Train Epoch: 6 [80000/160000 (50%)]	Loss: 0.671972
Train Epoch: 6 [100000/160000 (62%)]	Loss: 0.668719
Train Epoch: 6 [120000/160000 (75%)]	Loss: 0.664775
Train Epoch: 6 [140000/160000 (88%)]	Loss: 0.660152

Test set: Average loss: 0.6529, Accuracy: 29874/40000 (74.685%)

Train Epoch: 7 [0/160000 (0%)]	Loss: 0.666156
Train Epoch: 7 [20000/160000 (12%)]	Loss: 0.666693
Train Epoch: 7 [40000/160000 (25%)]	Loss: 0.664908
Train Epoch: 7 [60000/160000 (38%)]	Loss: 0.665601
Train Epoch: 7 [80000/160000 (50%)]	Loss: 0.658841
Train Epoch: 7 [100000/160000 (62%)]	Loss: 0.659382
Train Epoch: 7 [120000/160000 (75%)]	Loss: 0.662474
Train Epoch: 7 [140000/160000 (88%)]	Loss: 0.658091

Test set: Average loss: 0.6456, Accuracy: 30067/40000 (75.168%)

Train Epoch: 8 [0/160000 (0%)]	Loss: 0.657233
Train Epoch: 8 [20000/160000 (12%)]	Loss: 0.653101
Train Epoch: 8 [40000/160000 (25%)]	Loss: 0.660574
Train Epoch: 8 [60000/160000 (38%)]	Loss: 0.658713
Train Epoch: 8 [80000/160000 (50%)]	Loss: 0.657150
Train Epoch: 8 [100000/160000 (62%)]	Loss: 0.653666
Train Epoch: 8 [120000/160000 (75%)]	Loss: 0.658895
Train Epoch: 8 [140000/160000 (88%)]	Loss: 0.653125

Test set: Average loss: 0.6383, Accuracy: 30204/40000 (75.510%)

Train Epoch: 9 [0/160000 (0%)]	Loss: 0.654367
Train Epoch: 9 [20000/160000 (12%)]	Loss: 0.656965
Train Epoch: 9 [40000/160000 (25%)]	Loss: 0.652353
Train Epoch: 9 [60000/160000 (38%)]	Loss: 0.645441
Train Epoch: 9 [80000/160000 (50%)]	Loss: 0.649583
Train Epoch: 9 [100000/160000 (62%)]	Loss: 0.652022
Train Epoch: 9 [120000/160000 (75%)]	Loss: 0.648837
Train Epoch: 9 [140000/160000 (88%)]	Loss: 0.652718

Test set: Average loss: 0.6310, Accuracy: 30349/40000 (75.873%)

Train Epoch: 10 [0/160000 (0%)]	Loss: 0.638290
Train Epoch: 10 [20000/160000 (12%)]	Loss: 0.649517
Train Epoch: 10 [40000/160000 (25%)]	Loss: 0.646564
Train Epoch: 10 [60000/160000 (38%)]	Loss: 0.648319
Train Epoch: 10 [80000/160000 (50%)]	Loss: 0.645404
Train Epoch: 10 [100000/160000 (62%)]	Loss: 0.645872
Train Epoch: 10 [120000/160000 (75%)]	Loss: 0.640661
Train Epoch: 10 [140000/160000 (88%)]	Loss: 0.641092

Test set: Average loss: 0.6238, Accuracy: 30381/40000 (75.953%)


 training DNN with 200000 data points and SGD lr=0.010000. 

Train Epoch: 1 [0/160000 (0%)]	Loss: 0.696070
Train Epoch: 1 [20000/160000 (12%)]	Loss: 0.677516
Train Epoch: 1 [40000/160000 (25%)]	Loss: 0.675997
Train Epoch: 1 [60000/160000 (38%)]	Loss: 0.669214
Train Epoch: 1 [80000/160000 (50%)]	Loss: 0.656146
Train Epoch: 1 [100000/160000 (62%)]	Loss: 0.647163
Train Epoch: 1 [120000/160000 (75%)]	Loss: 0.638084
Train Epoch: 1 [140000/160000 (88%)]	Loss: 0.630936

Test set: Average loss: 0.6022, Accuracy: 29903/40000 (74.757%)

Train Epoch: 2 [0/160000 (0%)]	Loss: 0.628269
Train Epoch: 2 [20000/160000 (12%)]	Loss: 0.620329
Train Epoch: 2 [40000/160000 (25%)]	Loss: 0.608322
Train Epoch: 2 [60000/160000 (38%)]	Loss: 0.603806
Train Epoch: 2 [80000/160000 (50%)]	Loss: 0.598223
Train Epoch: 2 [100000/160000 (62%)]	Loss: 0.595937
Train Epoch: 2 [120000/160000 (75%)]	Loss: 0.576972
Train Epoch: 2 [140000/160000 (88%)]	Loss: 0.581402

Test set: Average loss: 0.5391, Accuracy: 30541/40000 (76.353%)

Train Epoch: 3 [0/160000 (0%)]	Loss: 0.576532
Train Epoch: 3 [20000/160000 (12%)]	Loss: 0.580321
Train Epoch: 3 [40000/160000 (25%)]	Loss: 0.560484
Train Epoch: 3 [60000/160000 (38%)]	Loss: 0.583032
Train Epoch: 3 [80000/160000 (50%)]	Loss: 0.554103
Train Epoch: 3 [100000/160000 (62%)]	Loss: 0.550453
Train Epoch: 3 [120000/160000 (75%)]	Loss: 0.541751
Train Epoch: 3 [140000/160000 (88%)]	Loss: 0.575952

Test set: Average loss: 0.5054, Accuracy: 30784/40000 (76.960%)

Train Epoch: 4 [0/160000 (0%)]	Loss: 0.558233
Train Epoch: 4 [20000/160000 (12%)]	Loss: 0.548893
Train Epoch: 4 [40000/160000 (25%)]	Loss: 0.548963
Train Epoch: 4 [60000/160000 (38%)]	Loss: 0.552454
Train Epoch: 4 [80000/160000 (50%)]	Loss: 0.551623
Train Epoch: 4 [100000/160000 (62%)]	Loss: 0.540242
Train Epoch: 4 [120000/160000 (75%)]	Loss: 0.547339
Train Epoch: 4 [140000/160000 (88%)]	Loss: 0.541507

Test set: Average loss: 0.4875, Accuracy: 31113/40000 (77.782%)

Train Epoch: 5 [0/160000 (0%)]	Loss: 0.522058
Train Epoch: 5 [20000/160000 (12%)]	Loss: 0.503705
Train Epoch: 5 [40000/160000 (25%)]	Loss: 0.531778
Train Epoch: 5 [60000/160000 (38%)]	Loss: 0.548663
Train Epoch: 5 [80000/160000 (50%)]	Loss: 0.514593
Train Epoch: 5 [100000/160000 (62%)]	Loss: 0.514267
Train Epoch: 5 [120000/160000 (75%)]	Loss: 0.517169
Train Epoch: 5 [140000/160000 (88%)]	Loss: 0.509606

Test set: Average loss: 0.4774, Accuracy: 31189/40000 (77.972%)

Train Epoch: 6 [0/160000 (0%)]	Loss: 0.542238
Train Epoch: 6 [20000/160000 (12%)]	Loss: 0.525145
Train Epoch: 6 [40000/160000 (25%)]	Loss: 0.509262
Train Epoch: 6 [60000/160000 (38%)]	Loss: 0.504582
Train Epoch: 6 [80000/160000 (50%)]	Loss: 0.518707
Train Epoch: 6 [100000/160000 (62%)]	Loss: 0.507325
Train Epoch: 6 [120000/160000 (75%)]	Loss: 0.497972
Train Epoch: 6 [140000/160000 (88%)]	Loss: 0.535714

Test set: Average loss: 0.4693, Accuracy: 31356/40000 (78.390%)

Train Epoch: 7 [0/160000 (0%)]	Loss: 0.522787
Train Epoch: 7 [20000/160000 (12%)]	Loss: 0.504788
Train Epoch: 7 [40000/160000 (25%)]	Loss: 0.520766
Train Epoch: 7 [60000/160000 (38%)]	Loss: 0.527349
Train Epoch: 7 [80000/160000 (50%)]	Loss: 0.515260
Train Epoch: 7 [100000/160000 (62%)]	Loss: 0.518895
Train Epoch: 7 [120000/160000 (75%)]	Loss: 0.503900
Train Epoch: 7 [140000/160000 (88%)]	Loss: 0.497610

Test set: Average loss: 0.4656, Accuracy: 31394/40000 (78.485%)

Train Epoch: 8 [0/160000 (0%)]	Loss: 0.514254
Train Epoch: 8 [20000/160000 (12%)]	Loss: 0.505731
Train Epoch: 8 [40000/160000 (25%)]	Loss: 0.502496
Train Epoch: 8 [60000/160000 (38%)]	Loss: 0.519354
Train Epoch: 8 [80000/160000 (50%)]	Loss: 0.509659
Train Epoch: 8 [100000/160000 (62%)]	Loss: 0.501614
Train Epoch: 8 [120000/160000 (75%)]	Loss: 0.512401
Train Epoch: 8 [140000/160000 (88%)]	Loss: 0.483391

Test set: Average loss: 0.4614, Accuracy: 31482/40000 (78.705%)

Train Epoch: 9 [0/160000 (0%)]	Loss: 0.504925
Train Epoch: 9 [20000/160000 (12%)]	Loss: 0.507819
Train Epoch: 9 [40000/160000 (25%)]	Loss: 0.503934
Train Epoch: 9 [60000/160000 (38%)]	Loss: 0.500493
Train Epoch: 9 [80000/160000 (50%)]	Loss: 0.501396
Train Epoch: 9 [100000/160000 (62%)]	Loss: 0.512802
Train Epoch: 9 [120000/160000 (75%)]	Loss: 0.495267
Train Epoch: 9 [140000/160000 (88%)]	Loss: 0.500724

Test set: Average loss: 0.4587, Accuracy: 31528/40000 (78.820%)

Train Epoch: 10 [0/160000 (0%)]	Loss: 0.487282
Train Epoch: 10 [20000/160000 (12%)]	Loss: 0.492254
Train Epoch: 10 [40000/160000 (25%)]	Loss: 0.491856
Train Epoch: 10 [60000/160000 (38%)]	Loss: 0.480748
Train Epoch: 10 [80000/160000 (50%)]	Loss: 0.516473
Train Epoch: 10 [100000/160000 (62%)]	Loss: 0.506555
Train Epoch: 10 [120000/160000 (75%)]	Loss: 0.484160
Train Epoch: 10 [140000/160000 (88%)]	Loss: 0.507923

Test set: Average loss: 0.4558, Accuracy: 31579/40000 (78.948%)


 training DNN with 200000 data points and SGD lr=0.100000. 

Train Epoch: 1 [0/160000 (0%)]	Loss: 0.700151
Train Epoch: 1 [20000/160000 (12%)]	Loss: 0.632980
Train Epoch: 1 [40000/160000 (25%)]	Loss: 0.575079
Train Epoch: 1 [60000/160000 (38%)]	Loss: 0.572152
Train Epoch: 1 [80000/160000 (50%)]	Loss: 0.537741
Train Epoch: 1 [100000/160000 (62%)]	Loss: 0.519731
Train Epoch: 1 [120000/160000 (75%)]	Loss: 0.506868
Train Epoch: 1 [140000/160000 (88%)]	Loss: 0.504988

Test set: Average loss: 0.4591, Accuracy: 31544/40000 (78.860%)

Train Epoch: 2 [0/160000 (0%)]	Loss: 0.487960
Train Epoch: 2 [20000/160000 (12%)]	Loss: 0.505924
Train Epoch: 2 [40000/160000 (25%)]	Loss: 0.476286
Train Epoch: 2 [60000/160000 (38%)]	Loss: 0.488260
Train Epoch: 2 [80000/160000 (50%)]	Loss: 0.477032
Train Epoch: 2 [100000/160000 (62%)]	Loss: 0.461733
Train Epoch: 2 [120000/160000 (75%)]	Loss: 0.482932
Train Epoch: 2 [140000/160000 (88%)]	Loss: 0.495323

Test set: Average loss: 0.4465, Accuracy: 31782/40000 (79.455%)

Train Epoch: 3 [0/160000 (0%)]	Loss: 0.461676
Train Epoch: 3 [20000/160000 (12%)]	Loss: 0.478750
Train Epoch: 3 [40000/160000 (25%)]	Loss: 0.474325
Train Epoch: 3 [60000/160000 (38%)]	Loss: 0.482127
Train Epoch: 3 [80000/160000 (50%)]	Loss: 0.463712
Train Epoch: 3 [100000/160000 (62%)]	Loss: 0.461602
Train Epoch: 3 [120000/160000 (75%)]	Loss: 0.466510
Train Epoch: 3 [140000/160000 (88%)]	Loss: 0.431422

Test set: Average loss: 0.4407, Accuracy: 31889/40000 (79.722%)

Train Epoch: 4 [0/160000 (0%)]	Loss: 0.454191
Train Epoch: 4 [20000/160000 (12%)]	Loss: 0.452877
Train Epoch: 4 [40000/160000 (25%)]	Loss: 0.470446
Train Epoch: 4 [60000/160000 (38%)]	Loss: 0.464326
Train Epoch: 4 [80000/160000 (50%)]	Loss: 0.487929
Train Epoch: 4 [100000/160000 (62%)]	Loss: 0.444073
Train Epoch: 4 [120000/160000 (75%)]	Loss: 0.469638
Train Epoch: 4 [140000/160000 (88%)]	Loss: 0.464015

Test set: Average loss: 0.4389, Accuracy: 31934/40000 (79.835%)

Train Epoch: 5 [0/160000 (0%)]	Loss: 0.469626
Train Epoch: 5 [20000/160000 (12%)]	Loss: 0.451672
Train Epoch: 5 [40000/160000 (25%)]	Loss: 0.459388
Train Epoch: 5 [60000/160000 (38%)]	Loss: 0.491337
Train Epoch: 5 [80000/160000 (50%)]	Loss: 0.454779
Train Epoch: 5 [100000/160000 (62%)]	Loss: 0.442819
Train Epoch: 5 [120000/160000 (75%)]	Loss: 0.486913
Train Epoch: 5 [140000/160000 (88%)]	Loss: 0.475788

Test set: Average loss: 0.4374, Accuracy: 31939/40000 (79.847%)

Train Epoch: 6 [0/160000 (0%)]	Loss: 0.440321
Train Epoch: 6 [20000/160000 (12%)]	Loss: 0.474917
Train Epoch: 6 [40000/160000 (25%)]	Loss: 0.451301
Train Epoch: 6 [60000/160000 (38%)]	Loss: 0.441167
Train Epoch: 6 [80000/160000 (50%)]	Loss: 0.463579
Train Epoch: 6 [100000/160000 (62%)]	Loss: 0.452635
Train Epoch: 6 [120000/160000 (75%)]	Loss: 0.452458
Train Epoch: 6 [140000/160000 (88%)]	Loss: 0.456863

Test set: Average loss: 0.4358, Accuracy: 31944/40000 (79.860%)

Train Epoch: 7 [0/160000 (0%)]	Loss: 0.459115
Train Epoch: 7 [20000/160000 (12%)]	Loss: 0.446554
Train Epoch: 7 [40000/160000 (25%)]	Loss: 0.440093
Train Epoch: 7 [60000/160000 (38%)]	Loss: 0.451953
Train Epoch: 7 [80000/160000 (50%)]	Loss: 0.458189
Train Epoch: 7 [100000/160000 (62%)]	Loss: 0.451847
Train Epoch: 7 [120000/160000 (75%)]	Loss: 0.450336
Train Epoch: 7 [140000/160000 (88%)]	Loss: 0.456441

Test set: Average loss: 0.4349, Accuracy: 31949/40000 (79.873%)

Train Epoch: 8 [0/160000 (0%)]	Loss: 0.439282
Train Epoch: 8 [20000/160000 (12%)]	Loss: 0.453577
Train Epoch: 8 [40000/160000 (25%)]	Loss: 0.463878
Train Epoch: 8 [60000/160000 (38%)]	Loss: 0.443377
Train Epoch: 8 [80000/160000 (50%)]	Loss: 0.451906
Train Epoch: 8 [100000/160000 (62%)]	Loss: 0.445123
Train Epoch: 8 [120000/160000 (75%)]	Loss: 0.458518
Train Epoch: 8 [140000/160000 (88%)]	Loss: 0.438410

Test set: Average loss: 0.4341, Accuracy: 31963/40000 (79.907%)

Train Epoch: 9 [0/160000 (0%)]	Loss: 0.461733
Train Epoch: 9 [20000/160000 (12%)]	Loss: 0.446442
Train Epoch: 9 [40000/160000 (25%)]	Loss: 0.462070
Train Epoch: 9 [60000/160000 (38%)]	Loss: 0.461662
Train Epoch: 9 [80000/160000 (50%)]	Loss: 0.446705
Train Epoch: 9 [100000/160000 (62%)]	Loss: 0.446212
Train Epoch: 9 [120000/160000 (75%)]	Loss: 0.429772
Train Epoch: 9 [140000/160000 (88%)]	Loss: 0.455330

Test set: Average loss: 0.4337, Accuracy: 31998/40000 (79.995%)

Train Epoch: 10 [0/160000 (0%)]	Loss: 0.432510
Train Epoch: 10 [20000/160000 (12%)]	Loss: 0.464081
Train Epoch: 10 [40000/160000 (25%)]	Loss: 0.426821
Train Epoch: 10 [60000/160000 (38%)]	Loss: 0.441955
Train Epoch: 10 [80000/160000 (50%)]	Loss: 0.433958
Train Epoch: 10 [100000/160000 (62%)]	Loss: 0.462393
Train Epoch: 10 [120000/160000 (75%)]	Loss: 0.441262
Train Epoch: 10 [140000/160000 (88%)]	Loss: 0.424818

Test set: Average loss: 0.4334, Accuracy: 31977/40000 (79.942%)