Custom data set training - low iou/accuracy

[reorder-cv]

Hi Lumi team,
For the last 4 weeks are running various iteration of training with our custom data set. The setup to run the training is good. Till date we had tried

1. Using Fastrcnn and SSD.
2. Training with one class of upto 320 images. We had gradually increased it form 80 to 320. (high quality images around 2mb in size, mostly consumable products which are found in super market)
3. Using csv file format and using xmin,xmax,ymin,ymax format for annotation.
4. Version
   Lumi : latest version
   Python : Python 3.6.5 :: Anaconda, Inc.
   Tensor flow : 1.10.0
   aws : ec2 Linux 4.4.0-1075-aws
5. using the recommended values in the config.yml
   learning_rate:
   decay_method: piecewise_constant

   Custom dataset for Luminoth Tutorial

   boundaries: [1000000,1200000]
   values: [0.0001,0.0001,0.00001]

Issue

1. Image prediction is very poor, the bounding boxes are over the images at around 2500 epochs and then start to degrade.
2. Loss is not getting consistently reduced as we progress the iteration. Accuracy is fluctuating and there is no pattern.
3. For testing we using only the images we had given in training, but still the test results are not good.

We can see a lot of potential with lumi, need your suggestion/hints on data set size / config file changes.

Thanks
Kani