This paper presents the results of a techno-economic investigation of a nanofiltration (NF) and reverse osmosis (RO) process for treating brackish water. Optimization experiments of six commercially available small scale RO and NF membranes were carried out using formulated artificial groundwater. A predictive model was developed by using response surface methodology (RSM) for optimization of input process parameters of brackish water membrane processes to simultaneously maximize water recovery and salt rejection while minimizing energy demand. A predictive model using multiple response optimization revealed that CSM RO and NF250 membranes showed the optimal efficiency with 20.24% and 18.98% water recovery, 90.22% and 70.64% salt rejection and 17.87 kWh/m3 and 9.35 kWh/m3 of specific energy consumption (SEC), respectively. Furthermore, confirmation of RSM predictions was carried out by an artificial neural network (ANN) model trained by RSM experimental data. Predicted values by both RSM and ANN modeling methodologies were compared and found within the acceptable range. Finally, a membrane validation experiment was carried out successfully at proposed optimal conditions, which proved the accuracy of the employed RSM and ANN models. Detailed analyses of the economic assessment showed that the recovery rate can play a major role in reducing the cost of a membrane system.