The separation membranes, constructed with graphene and its derivatives (e.g., graphene oxide) as the building blocks, have unique laminar structures and tunable physicochemical properties, and therefore show promising applications in many areas such as water desalination, wastewater treatment and pervaporation. Based on their microstructure, the graphene-based membranes are classified into two typical forms: single-layered nanoporous graphene films and laminated graphene-oxide-based membranes. In this paper, the structural features and fabrication methods of both two types of membranes are briefly introduced, their mass transport behaviors and mechanisms are thoroughly discussed based on theoretical simulations and experimental results, and the recent progresses on their water treatment applications are reviewed with an emphasis on the structural modulation and property optimization of laminated graphene-oxide-based membranes. Finally, the research trends and challenges in the field of graphene-based separation membranes are discussed.