Glaucoma, an abnormality in the eye condition that, if not treated within safe time limit, can result in visual loss. Glaucoma diagnosis requires professionals to identify minor structural changes in the structure of the optic disc and optic cup from retinal fundus images in a short period of time. Deep learning algorithms have been employed effectively in the segmentation of biomedical images and the identification of disease. To accomplish good generalization, model training requires comprehensive annotations, which is a difficult task. The intended objective of the present study is to come up with and train an distinctive multi-task deep learning model for automated fundus image segmentation and classification. The multitask model learns for segmentation task of Optic Disc (OD) and Optic Cup (OC) and classification task for accurate glaucoma detection using both structural and image based features. The multi-task model proposed a modified U-net architecture in which Mobile-Netv2 is used in the encoder part, Graph Convolution Network (GCN) is used in the decoder part, and an attention module (AM) is used to locate the region of interest (ROI) for better feature extraction. The implementation of this model is done using three fundus image datasets such as ORIGA, REFUGE, and DRISTI-GS. The performance of the proposed multi-task model is compared with some existing methods and shows maximum accuracy of 97.43% and AUROC of 0.985 for glaucoma detection task and high quality OD and OC segmented images with dice coefficient of 97.95% and 96.11% respectively for segmentation task.