The test accuracy is not changed with different training settings

[ChiaoThon]

Hi Snwagh,
In globals.h at /src file, after I change the #define MNIST false into #define MNIST true, NUM_LAYERS to 4, LOG_MINI_BATCH to 0 in #if MNIST frame, NO_OF_EPOCHS to 1, NUM_ITERATIONS to 1, then at secondary.cpp, I change for (int i = 0; i < NUM_ITERATIONS; ++i) in void test(NeuralNetwork* net) function to for (int i = 0; i < 10000; ++i). Finally, I change the network to be secureml in main.cpp and train the network with make standalone and do the test by changing

whichNetwork += " train";
train(network, config);

into

whichNetwork += " test";
test(network);

it gives me 99% accuracy which seems not correct as I just use 1 iteration (which means one piece of training data) to train the network, I changed the number of iteration but always get the same accuracy, can you give some tips for it or did I test it in wrong way?

Thanks very much!

[snwagh]

I notice a few issues which I'll point to shortly: First, might be because you don't have the datasets (they're not on Github). You'll have to download the MNIST dataset from here or some source and then parse it to be compatible with SecureNN (you can use the code inside mnist/ for parsing).

Secondly, in the current version of code, you'll have to run the training and the testing in the same run (otherwise there is no trained model for inference). Alternatively, you can train the model in standalone but will have to export the trained model to file and then import it for the testing (this importing and exporting is not currently implemented but shouldn't be too difficult).

Finally, I also believe in the current version you should run make mnistSA instead of make standalone. You can verify the command line inputs from the makefile to double check the datasets for the run (I'm travelling with limited/no internet over the next few days but will look into this in more detail once I'm back).

[ChiaoThon]

Hi snwagh,

I tried to parse the whole dataset using the tools in mnist/ but failed, can you explain a little bit on how to successfully parse the whole dataset given the four downloaded data (train-images-idx3-ubyte.gz, train-labels-idx1-ubyte.gz, t10k-images-idx3-ubyte.gz and t10k-labels-idx1-ubyte.gz) from here? BTW, I think the code needs the .csv file from here to parse correctly.

[snwagh]

You're right, the parser is meant for the .csv files from your above link. All you will need to do is make the file and run it with the the paths to the two csv files as arguments. Alternatively, edit the file paths in the run.sh file and it should generate the 4 files for you.

[ChiaoThon]

Hi snwagh,

After I load the whole data set, I did the same modification as mentioned at very beginning, and train the network with make mnistSA and do the test by

whichNetwork += " train";
train(network, config);
whichNetwork += " test";
test(network);

in the main.cpp but I get 10% accuracy. the accuracy remains with several tests. Is the system not fit for batchsize with 1?

BTW, I also copy the parsed mnist data in the claimed folder /files/MNIST/

[snwagh]

For ML reasons, I think a batch size of 1 doesn't train well (FYI, I have not tried training it with batch size 1). It is also the reason why you need stochastic gradient descent (averaging out the gradients in a mini-batch gives a smoother convergence). You might have to spend some more time tuning the parameters if you use a batch size of 1.

Also, for the simplistic network, all the hyper parameters (number of iterations, learning rate, initial weights and bias range) make a difference. A 10% accuracy network implies that your trained weights are essentially random or fixed classification (in my experience the latter). If you need deeper probing into the code, try printing out the outputs and use that as a starting point for debugging.

[0xlimE]

@ChiaoThon Did you ever get results with good accuracies from the standalone run? If so can you share your hyperparameters? Thanks a lot.

[zfscgy]

@ChiaoThon Did you ever get results with good accuracies from the standalone run? If so can you share your hyperparameters? Thanks a lot.

In the MnistSA setting, the accuracy is always around 10%. And in the 4PC setting, the predictions are totally wrong.

[ChiaoThon]

I have the same problem with the accuracy and the code just makes sure is it runnable, the accuracy is not guaranteed due to approximation calculation in backpropagation or hyperparameter or others ? @zfscgy @0xlimE @snwagh

[snwagh]

The 4PC certainly is more legacy and needs a lot more testing and bug fixing. SA should also need testing/debugging but should be considerably easier to debug.

Accuracy 10% usually in my experience playing with the underlying number is when the NN effectively predicts randomly (MNIST has 10 classes to about 9-11% accuracy is generally seen in practice).

The training does not work out of the box in practice. You need to tweak the hyperparameters (initial range of weights, learning rate etc.) but hopefully with a few hours of tweaking (and even with just a few forward-backward passes) you should see non-trivial learning (60-80% accuracy). Initialize the weights to around two or three orders below 1 as the simple networks have not way to protect against value sticking to either extremes.

For 4PC issues, try to refer to the correspionding 3PC code, it is better tested. For issues with the 3PC code, see if the corresponding functions in the Falcon codebase are useful for reference. I do believe the Mat-Mul, ReLU are well tested (and that's all you need for some of the networks). But if you suspect specific issues raise a git-issue.