safire/safire.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "safire.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [
        "Ov-m1nFU9jry",
        "1FV-VHZsLkhG",
        "WmnTaw3f9DF3",
        "2NcemUMdp92X",
        "0882m2pipfVU",
        "FEwTp1m6pA_-",
        "eojC1KSyFlf3"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/88lex/safire/blob/master/safire.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ov-m1nFU9jry",
        "colab_type": "text"
      },
      "source": [
        "### Safire"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FV-VHZsLkhG",
        "colab_type": "text"
      },
      "source": [
        "###### Instructions\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guMoYIjjMjgL",
        "colab_type": "text"
      },
      "source": [
        "1. If using Colab:\n",
        "  - Save copy of the ipynb file to your drive (Colab Notebooks folder)\n",
        "  - Run the init code cell to auth to your My Drive account. This gives you access to Drive where you will store your creds, svcaccts and xlsx/csv files.\n",
        "\n",
        "2. If Running locally then change the config to local folders\n",
        "3. Run the Config cell once to create empty /creds, /data and /svcaccts folders in the /safire folder if they do not exist.\n",
        "4. If you have creds.json and token files, drag them to the /safire/creds you just created.\n",
        "5. If you do not have creds.json and token files then create a creds.json file and drop it in safire/creds. Then run `auth projects` and `auth groups` (see [github](https://github.com/88lex/safire) for details.\n",
        "\n",
        "^^ All of this you do once, after which you will have a structure that looks like this:\n",
        "```\n",
        "/content/drive/My Drive\n",
        "  /safire\n",
        "    /creds\n",
        "      /creds.json\n",
        "      /token.pickle\n",
        "      /grptoken.pickle\n",
        "    /data\n",
        "    /svcaccts\n",
        "```\n",
        "\n",
        "After creating your credentials and folders:\n",
        "6. Enter one or more commands in the Commands section (empty cells are ignored). Command syntax is:\n",
        "  - `list projects`\n",
        "  - `list all`\n",
        "  - `list counts` will show you item counts for projects, drives, groups, etc.\n",
        "  - `add projects` will add the number of projects using a prefix that you entered in the config file\n",
        "  - `add projects 5 1 myproj` will create 5 projects starting with number 0001 using a prefix of myproj. Result: myproj0001, myproj0002 .. myproj0005\n",
        "\n",
        "7. Run commands using `Runtime - Run All` or `Ctrl-F9`\n",
        "  - NOTE: If you run only the Command cell it will do nothing. You need to Run All in order to run all of the code.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkhRMr8DliCs",
        "colab_type": "text"
      },
      "source": [
        "###### **Init**\n",
        "\n",
        "- Init needs to be run once each time the Colab instance or Jupyter notebook is started\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e2xK3xi2MlXs",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "import os, importlib, sys, uuid, pickle\n",
        "import pandas as pd\n",
        "from base64 import b64decode\n",
        "from glob import glob\n",
        "from json import loads\n",
        "from time import sleep\n",
        "from pathlib import Path\n",
        "from google.auth.transport.requests import Request\n",
        "from google_auth_oauthlib.flow import InstalledAppFlow\n",
        "from googleapiclient.discovery import build\n",
        "from shutil import copyfile\n",
        "\n",
        "if importlib.util.find_spec('fire') is None: os.system('pip install fire')\n",
        "import fire\n",
        "\n",
        "#@markdown Root dir when using Colab GDrive mount(recommended to leave default):\n",
        "try:\n",
        "    from google.colab import drive\n",
        "    gdrive_mount = '/content/drive' #@param ['/content/drive'] {allow-input: true}\n",
        "    gd_output_dir = '/My Drive/safire' #@param ['/My Drive/safire'] {allow-input: true}\n",
        "    drive.mount(gdrive_mount)\n",
        "    safire_dir = gdrive_mount + gd_output_dir\n",
        "except:\n",
        "    base_path = str(Path.home())\n",
        "    safire_dir = os.path.join(base_path, \"safire\")\n",
        "    os.makedirs(safire_dir, exist_ok=True)\n",
        "    cfg_file = os.path.join(safire_dir, \"config.py\")\n",
        "    if os.path.exists(cfg_file):\n",
        "        copyfile(cfg_file, \"config.py\")\n",
        "    else:\n",
        "        copyfile(\"default_config.py\", \"config.py\")\n",
        "        copyfile(\"default_config.py\", cfg_file)\n",
        "\n",
        "    ## @markdown Root directory when safire running locally (dropdown choices):\n",
        "    # local_safire_dir = '~/safire' #@param ['~/safire', '/opt/safire/safire', '~/../opt/safire/safire', \"safire\"] {allow-input: true}\n",
        "    # safire_dir = local_safire_dir\n",
        "# @markdown External config file located in local_safire_dir. Default: config.py:\n",
        "use_external_config = True  # @param {type:\"boolean\"}\n",
        "ext_conf_file = \"config.py\"  # @param [\"config.py\"] {allow-input: true}\n",
        "ext_conf_file = os.path.join(safire_dir, ext_conf_file)\n",
        "\n",
        "\n",
        "# except:\n",
        "#     #@markdown Root directory when safire running locally (dropdown choices):\n",
        "#     local_safire_dir = '~/safire' #@param ['~/safire', '/opt/safire/safire', '~/../opt/safire/safire', \"safire\"] {allow-input: true}\n",
        "#     safire_dir = local_safire_dir\n",
        "# #@markdown External config file located in local_safire_dir. Default: config.py:\n",
        "# use_external_config = True #@param {type:\"boolean\"}\n",
        "# ext_conf_file = \"/config.py\" #@param [\"/config.py\"] {allow-input: true}\n",
        "# ext_conf_file = safire_dir + ext_conf_file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WmnTaw3f9DF3",
        "colab_type": "text"
      },
      "source": [
        "###### Config\n",
        "- Tells safire where to read and write files, how many 'things' to create and what to call them"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KEdQk5ai8JTW",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import random\n",
        "import string\n",
        "from pathlib import Path\n",
        "from time import sleep\n",
        "\n",
        "#@markdown Project prefix must be unique across all of Google. If blank a random project prefix will be generated\n",
        "#@markdown :  NOTE - DO NOT keep sample name below = `project`\n",
        "project_prefix = \"project\"        #@param [ \"\"] {allow-input: true}\n",
        "if not project_prefix:\n",
        "    letters = string.ascii_lowercase\n",
        "    project_prefix = ''.join(random.choice(letters) for i in range(10))\n",
        "    print(f\"Project prefix = {project_prefix}\")\n",
        "\n",
        "#@markdown Prefix for service account emails is =required= (can be anything you like)\n",
        "email_prefix = \"sa\" #@param [ \"sa\", \"svcacc\", \"bananas\"] {allow-input: true}\n",
        "\n",
        "#@markdown Prefix for JSON keys is =optional=\n",
        "json_prefix = \"\"          #@param {type:\"string\"}\n",
        "#@markdown Enter a group email address if using group functions (e.g. mygroup@domain.com)\n",
        "group_address = \"\"        #@param [ \"\", \"mygroup@domain.com\"] {allow-input: true}\n",
        "\n",
        "#@markdown Slide bars and dropdowns can be adjusted with arrow keys\n",
        "num_new_projects = 1      #@param {type:\"slider\", min:1, max:200, step:1}\n",
        "sas_per_project = 100     #@param {type:\"slider\", min:1, max:100, step:1}\n",
        "# max_projects =  100     #@param {type:\"integer\"} \n",
        "next_project_num = 1      #@param {type:\"integer\"}\n",
        "next_sa_num = 1           #@param {type:\"integer\"}\n",
        "#@markdown Note: JSON numbering can be independent of SA numbering\n",
        "next_json_num = 1         #@param {type:\"integer\"}\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "# #@markdown Enter 'safire_dir'where creds, SAs and data files will be saved. Default is \n",
        "# #@markdown `/content/drive/My Drive/safire`\n",
        "# safire_dir = \"/content/drive/My Drive/safire\"#@param [\"/content/drive/My Drive/safire\"] {allow-input: true}\n",
        "\n",
        "#@markdown These are subfolders of 'safire_dir' above. \n",
        "#@markdown Creds and tokens are stored in the same folder\n",
        "cred_folder = \"/creds\"       #@param {type:\"string\"}\n",
        "cred_path = safire_dir + cred_folder\n",
        "cred_file = \"/creds.json\"    #@param {type:\"string\"}\n",
        "credentials = cred_path + cred_file\n",
        "token_file = \"/token.pickle\" #@param {type:\"string\"}\n",
        "token = f\"{cred_path}/token.pickle\"\n",
        "sa_folder = \"/svcaccts\"      #@param {type:\"string\"}\n",
        "sa_path = safire_dir + sa_folder\n",
        "data_folder = \"/data\"        #@param {type:\"string\"}\n",
        "data_path = safire_dir + data_folder\n",
        "\n",
        "#@markdown Create and store a separate token for groups. You can use the same creds\n",
        "group_cred_file = \"/creds.json\" #@param {type:\"string\"}\n",
        "group_credentials = cred_path + group_cred_file\n",
        "group_token_file = \"/grptoken.pickle\" #@param {type:\"string\"}\n",
        "group_token = cred_path + group_token_file\n",
        "\n",
        "#@markdown ___\n",
        "\n",
        "#@markdown Set zero padding for names. e.g. jpad=6 yields 000001.json. Set to 1 if no prefix needed.\n",
        "ppad = 4             #@param {type:\"integer\"}\n",
        "spad = 6             #@param {type:\"integer\"}\n",
        "jpad = 6             #@param {type:\"integer\"}\n",
        "#@markdown retry and sleep_time help safire deal with slow google apis. Adjust as needed\n",
        "retry =            5#@param {type:\"integer\"}\n",
        "sleep_time =        5#@param {type:\"integer\"}\n",
        "\n",
        "## #@markdown [API Service Names and Versions]\n",
        "DRIVE = [\"drive\", \"v3\"]\n",
        "CLOUD = [\"cloudresourcemanager\", \"v1\"]\n",
        "IAM = [\"iam\", \"v1\"]\n",
        "ADMIN = [\"admin\", \"directory_v1\"]\n",
        "SVCUSAGE = [\"serviceusage\", \"v1\"]\n",
        "SHEETS = [\"sheets\", \"v4\"]\n",
        "svcs_to_enable = [\"iam\", \"drive\"]\n",
        "\n",
        "### Generate empty directories if they do not exist\n",
        "if not os.path.isdir(safire_dir):\n",
        "    os.makedirs(safire_dir, exist_ok=True)\n",
        "    sleep(sleep_time)\n",
        "if not os.path.isdir(sa_path):\n",
        "    os.makedirs(sa_path, exist_ok=True)\n",
        "if not os.path.isdir(cred_path):\n",
        "    os.makedirs(cred_path, exist_ok=True)\n",
        "if not os.path.isdir(data_path):\n",
        "    os.makedirs(data_path, exist_ok=True)\n",
        "\n",
        "if use_external_config:\n",
        "    try:\n",
        "        exec(open(ext_conf_file).read())\n",
        "    except:\n",
        "        print(f\"Config file = {ext_conf_file} not found, not executed\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tjDqfHn89Ilg",
        "colab_type": "text"
      },
      "source": [
        "###### Utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "67Ezfluw83Gb",
        "colab_type": "text"
      },
      "source": [
        "This section conatins the utils functions then main commands/subcommands"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2HM3pfrX86w0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import pickle\n",
        "import uuid\n",
        "import pandas as pd\n",
        "\n",
        "from glob import glob\n",
        "from pathlib import Path\n",
        "from time import sleep\n",
        "\n",
        "from google.auth.transport.requests import Request\n",
        "from google_auth_oauthlib.flow import InstalledAppFlow\n",
        "from googleapiclient.discovery import build\n",
        "\n",
        "\n",
        "class Help:\n",
        "    \"\"\"These small functions support repeated activities in other classes/functions\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Help, self).__init__()\n",
        "\n",
        "    def _svc(self, service, version, tkn):\n",
        "        try:\n",
        "            with open(tkn, \"rb\") as t:\n",
        "                creds = pickle.load(t)\n",
        "            return build(service, version, credentials=creds)\n",
        "        except:\n",
        "            print(\"No valid token found\")\n",
        "            auth = Auth()\n",
        "            auth.check()\n",
        "\n",
        "    def _export(self, dset, filter, file_tag, fields, ltype, prt=True):\n",
        "        df = pd.DataFrame(dset)\n",
        "        fname = f\"{ltype}_list_{filter}_{file_tag}\"\n",
        "        pd.set_option('display.max_rows', None)\n",
        "        df.to_csv(\"{}/{}.csv\".format(data_path, fname))\n",
        "        df.to_excel(\"{}/{}.xlsx\".format(data_path, fname))\n",
        "        if prt:\n",
        "            print(df[fields])\n",
        "            print(f\"  {len(df)} {ltype} found\")\n",
        "            print(f\"\\nData for {ltype} exported to {fname}.csv and .xlsx in folder /{data_path}\\n\")\n",
        "\n",
        "    def _print1(self, plist, ltype):\n",
        "        print('', *plist, sep='\\n')\n",
        "        print(f\"  {len(plist)} {ltype} found\")\n",
        "\n",
        "    def _pre_pad(self, prefix, pad, number):\n",
        "        return prefix + (('0' * pad) + str(number))[-pad:]\n",
        "\n",
        "\n",
        "class BatchJob():\n",
        "\n",
        "    def __init__(self, service):\n",
        "        self.batch = service.new_batch_http_request(callback=self.callback_handler)\n",
        "        self.batch_resp = []\n",
        "\n",
        "    def add(self, to_add, request_id=None):\n",
        "        self.batch.add(to_add, request_id=request_id)\n",
        "\n",
        "    def callback_handler(self, rid, resp, exception):\n",
        "        response = {'request_id': rid, 'exception': None, 'response': None}\n",
        "        if exception is not None:\n",
        "            response['exception'] = exception\n",
        "        else:\n",
        "            response['response'] = resp\n",
        "        self.batch_resp.append(response)\n",
        "\n",
        "    def execute(self):\n",
        "        try:\n",
        "            self.batch.execute()\n",
        "        except:\n",
        "            pass\n",
        "        return self.batch_resp\n",
        "\n",
        "\n",
        "class Auth:\n",
        "    \"\"\"Authorize the app to access your projects, SAs, drives and groups. To generate creds.json go to\n",
        "    https://developers.google.com/apps-script/api/quickstart/python , click Enable then download a json,\n",
        "    rename it to creds.json and put a copy in the /creds folder\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Auth, self).__init__()\n",
        "        self.scopes_proj = [\n",
        "            \"https://www.googleapis.com/auth/drive\",\n",
        "            \"https://www.googleapis.com/auth/cloud-platform\",\n",
        "            \"https://www.googleapis.com/auth/iam\",\n",
        "        ]\n",
        "            # \"https://www.googleapis.com/auth/spreadsheets\",\n",
        "\n",
        "        self.scopes_group = [\n",
        "            \"https://www.googleapis.com/auth/admin.directory.group\",\n",
        "            \"https://www.googleapis.com/auth/admin.directory.group.member\",\n",
        "        ]\n",
        "        self.scopes_all = self.scopes_proj + self.scopes_group\n",
        "\n",
        "    def ask(self):\n",
        "        pass\n",
        "\n",
        "    def check(self):\n",
        "        filelist = [credentials, token, group_credentials, group_token]\n",
        "        file_exists = [os.path.isfile(i) for i in filelist]\n",
        "        [print(f\"File = {i[0]} exists = {i[1]}\") for i in zip(filelist, file_exists)]\n",
        "        if not file_exists[0]:\n",
        "            print(\n",
        "                f\"Credentials file is missing. Download from Google console and run 'auth'\"\n",
        "            )\n",
        "        if not file_exists[2]:\n",
        "            print(\n",
        "                f\"Group credentials file is missing. Download from Google console and run 'auth'\"\n",
        "                f\"Note that the group credentials file is normally the same as the main projects credentials\"\n",
        "                f\"file - But you can optionally use separate credentials files. Specify in config.py\"\n",
        "            )\n",
        "        yes_no = input(\"Generate token for projects, SAs, drives? y/n: \")\n",
        "        if yes_no.lower() in [\"y\", \"yes\"]:\n",
        "            self.projects(credentials, token)\n",
        "        gyes_no = input(\"Generate token for groups? y/n: \")\n",
        "        if gyes_no.lower() in [\"y\", \"yes\"]:\n",
        "            self.groups(group_credentials, group_token)\n",
        "        exit()\n",
        "\n",
        "    def projects(self, credentials=credentials, token=token):\n",
        "        \"\"\"Create an auth token for accessing and changing projects,\n",
        "        service accounts, json keys and drives\"\"\"\n",
        "        self.get_token(credentials, self.scopes_proj, token)\n",
        "\n",
        "    def groups(self, credentials=credentials, group_token=group_token):\n",
        "        \"\"\"Create an auth token for adding/removing group members\"\"\"\n",
        "        self.get_token(credentials, self.scopes_group, group_token)\n",
        "\n",
        "    def all(self, credentials=credentials, token=token):\n",
        "        \"\"\"Create an auth token with APIs enabled for projects and groups\"\"\"\n",
        "        self.projects(credentials, token)\n",
        "        self.groups(group_credentials, group_token)\n",
        "\n",
        "    def get_token(self, credentials, scopes, token):\n",
        "        if not credentials or os.path.isfile(credentials[0]):\n",
        "            print(\n",
        "                f\"Credentials file is missing. Download from Google console and save as {credentials}\"\n",
        "            )\n",
        "            exit()\n",
        "        cred_file = glob(credentials)\n",
        "        creds = None\n",
        "        if os.path.exists(token):\n",
        "            with open(token, \"rb\") as tkn:\n",
        "                creds = pickle.load(tkn)\n",
        "        if not creds or not creds.valid:\n",
        "            if creds and creds.expired and creds.refresh_token:\n",
        "                creds.refresh(Request())\n",
        "            else:\n",
        "                flow = InstalledAppFlow.from_client_secrets_file(cred_file[0], scopes)\n",
        "                yes_no = input(\n",
        "                    \"Run local server with browser? If no, generate console link. y/n: \"\n",
        "                )\n",
        "                if yes_no.lower() in [\"y\", \"yes\"]:\n",
        "                    creds = flow.run_local_server()\n",
        "                else:\n",
        "                    creds = flow.run_console()\n",
        "            with open(token, \"wb\") as tkn:\n",
        "                print(\"Writing/updating token\")\n",
        "                pickle.dump(creds, tkn)\n",
        "        else:\n",
        "            print(\"Credentials and token exist and appear to be valid\")\n",
        "            print(f\"credentials = {cred_file[0]} and token = {token}\")\n",
        "            yes_no = input(\n",
        "                f\"Do you want to delete and regenerate your token file = {token}? y/n: \"\n",
        "            )\n",
        "            if yes_no.lower() in [\"y\", \"yes\"]:\n",
        "                os.remove(token)\n",
        "                self.get_token(credentials, scopes, token)\n",
        "            else:\n",
        "                print(\"Finished without creating new token\")\n",
        "\n",
        "\n",
        "class Link():\n",
        "    \"\"\"Create a symlink between safire's directories and another location\"\"\"\n",
        "\n",
        "    def dirs(self):\n",
        "        cwd = os.path.dirname(os.path.realpath(__file__))\n",
        "        dest = f\"{str(Path.home())}/safire\"\n",
        "        dest1 = input(f\"Choose dir to link. To keep default = {dest} press Enter: \")\n",
        "        if dest1:\n",
        "            dest = dest1\n",
        "        if os.path.exists(dest):\n",
        "            print(f\"Directory {dest} already exists. Exiting.\")\n",
        "            exit()\n",
        "        print(f\"Linking {cwd} to {dest}\")\n",
        "        if os.path.isdir(dest):\n",
        "            os.unlink(dest)\n",
        "        os.symlink(cwd, dest)\n",
        "\n",
        "\n",
        "class Commands:\n",
        "    \"\"\"safire creates projects, service accounts(SAs), json keys for SAs and adds SAs as group members.\\n Type -h\n",
        "    or --help after any command for info.  Usage: './safire list projects' or 'safire add members mygroup@domain.com'\n",
        "    Most commands accept a 'filter'to process subsets. If no filter all items are listed.\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        # Init()\n",
        "        self.list = List()\n",
        "        self.add = Add()\n",
        "        self.remove = Remove()\n",
        "        self.auth = Auth()\n",
        "        self.rename = Rename()\n",
        "        self.link = Link()\n",
        "        # alias commands for ease of use. e.g. 'safire add projects' = 'safire create projects'\n",
        "        self.create = self.add\n",
        "        self.enable = self.add\n",
        "        self.delete = self.remove\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouhOikY1qEZG",
        "colab_type": "text"
      },
      "source": [
        "###### List"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Hs3kkatqDOg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class List(Help):\n",
        "    \"\"\"List drives, projects, service accounts (SAs), SA json keys, groups and group members. \n",
        "    In most cases a filter can be applied.\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(List, self).__init__()\n",
        "\n",
        "    def drives(self, filter=\"\", file_tag=\"\", token=token, prt=True):\n",
        "        \"\"\"List team/shared drives. Match 'filter'\"\"\"\n",
        "        drivelist = []\n",
        "        resp = {\"nextPageToken\": None}\n",
        "        drive = self._svc(*DRIVE, token)\n",
        "        while \"nextPageToken\" in resp:\n",
        "            resp = (\n",
        "                drive.drives()\n",
        "                .list(\n",
        "                    fields=\"nextPageToken, drives(id,name)\",\n",
        "                    pageSize=100,\n",
        "                    pageToken=resp[\"nextPageToken\"],\n",
        "                )\n",
        "                .execute()\n",
        "            )\n",
        "            drivelist += resp[\"drives\"]\n",
        "        drivelist = [i for i in drivelist if str(filter) in i[\"name\"]]\n",
        "        if not prt:\n",
        "            return drivelist\n",
        "        self._export(drivelist, filter, file_tag, [\"id\", \"name\"], \"drives\", prt)\n",
        "\n",
        "    def projects(self, filter=\"\", file_tag=\"\", token=token, prt=True, exact=False):\n",
        "        \"\"\"List projects. Match 'filter'\"\"\"\n",
        "        cloud = self._svc(*CLOUD, token)\n",
        "        plist = cloud.projects().list().execute()[\"projects\"]\n",
        "        plist = [i for i in plist if str(filter) in i[\"projectId\"]]\n",
        "        if exact:\n",
        "            plist = [i for i in plist if str(filter) == i[\"projectId\"]]\n",
        "        if not prt:\n",
        "            return [i[\"projectId\"] for i in plist]\n",
        "        self._export(\n",
        "            plist,\n",
        "            filter,\n",
        "            file_tag,\n",
        "            [\"projectNumber\", \"projectId\", \"name\"],\n",
        "            \"projects\",\n",
        "            prt,\n",
        "        )\n",
        "\n",
        "    def sas(self, filter=\"\", exact=False, file_tag=\"\", token=token, prt=True):\n",
        "        \"\"\"List service accounts for projects. Projects match 'filter'.\n",
        "        Use '--exact=True' flag for exact project match.\"\"\"\n",
        "        svcacctlist, sa_summary = [], []\n",
        "        projId_list = self.projects(filter, file_tag, token, False, exact)\n",
        "        # if exact:\n",
        "        #     projId_list = [i for i in projId_list if i == filter]\n",
        "        iam = self._svc(*IAM, token)\n",
        "        for project in sorted(projId_list):\n",
        "            try:\n",
        "                resp = (\n",
        "                    iam.projects()\n",
        "                    .serviceAccounts()\n",
        "                    .list(name=\"projects/\" + project, pageSize=100)\n",
        "                    .execute()\n",
        "                )[\"accounts\"]\n",
        "                svcacctlist += resp\n",
        "                sa_summary += [\n",
        "                    f\"  {len(resp)} service accounts (SAs) found in project: {project}\"\n",
        "                ]\n",
        "            except:\n",
        "                sa_summary += [\n",
        "                    f\"  No service accounts (SAs) found in project: {project}\"\n",
        "                ]\n",
        "                continue\n",
        "        svcacctlist.sort(key=lambda x: x.get(\"email\"))\n",
        "        if not prt:\n",
        "            return (\n",
        "                [i[\"email\"] for i in svcacctlist],\n",
        "                [i[\"uniqueId\"] for i in svcacctlist],\n",
        "            )\n",
        "        if svcacctlist:\n",
        "            self._export(svcacctlist, filter, file_tag, [\"email\"], \"svc_accts\")\n",
        "        print(\"\\nSummary:\", *sa_summary, sep=\"\\n\")\n",
        "\n",
        "    def groups(self, filter=\"\", file_tag=\"\", group_token=group_token, prt=True):\n",
        "        \"\"\"List groups in the authorized account. Match 'filter'\"\"\"\n",
        "        svc = self._svc(*ADMIN, group_token)\n",
        "        grouplist = svc.groups().list(customer=\"my_customer\").execute()[\"groups\"]\n",
        "        grouplist = [i for i in grouplist if str(filter) in i[\"email\"]]\n",
        "        if not prt:\n",
        "            return [i[\"email\"] for i in grouplist]\n",
        "        self._export(\n",
        "            grouplist,\n",
        "            filter,\n",
        "            file_tag,\n",
        "            [\"id\", \"email\", \"directMembersCount\"],\n",
        "            \"groups\",\n",
        "            prt,\n",
        "        )\n",
        "\n",
        "    def members(self, filter=\"\", file_tag=\"\", group_token=group_token, prt=True):\n",
        "        \"\"\"List members in groups. Groups match 'filter'\"\"\"\n",
        "        memberslist, member_summary = [], []\n",
        "        group_list = self.groups(filter, file_tag, group_token, False)\n",
        "        svc = self._svc(*ADMIN, group_token)\n",
        "        for group in group_list:\n",
        "            try:\n",
        "                response = []\n",
        "                resp = {\"nextPageToken\": None}\n",
        "                while \"nextPageToken\" in resp:\n",
        "                    resp = (\n",
        "                        svc.members()\n",
        "                        .list(groupKey=group, pageToken=resp[\"nextPageToken\"])\n",
        "                        .execute()\n",
        "                    )\n",
        "                    response += resp[\"members\"]\n",
        "                memberslist += response\n",
        "            except:\n",
        "                member_summary += [f\"  No members found in group: {group}\"]\n",
        "                continue\n",
        "            member_summary += [f\"  {len(response)} members found in group: {group}\"]\n",
        "        if not prt:\n",
        "            return sorted([i[\"email\"] for i in memberslist if i[\"role\"] != \"OWNER\"])\n",
        "        self._export(memberslist, filter, file_tag, [\"email\", \"role\"], \"members\")\n",
        "        print(\"\\nSummary:\", *member_summary, sep=\"\\n\")\n",
        "\n",
        "    def jsons(self, sa_path=sa_path, filter=\"\", file_tag=\"\", prt=True):\n",
        "        \"\"\"alias: jsons = keys. List service account jsons/keys in the svcaccts folder. Match 'filter'\"\"\"\n",
        "        keys = glob(\"%s/*.json\" % sa_path)\n",
        "        json_keys = []\n",
        "        [json_keys.append(loads(open(key, \"r\").read())) for key in keys]\n",
        "        if filter:\n",
        "            json_keys = [key for key in json_keys if str(filter) in key[\"project_id\"]]\n",
        "        if prt and json_keys:\n",
        "            self._export(\n",
        "                json_keys, filter, file_tag, [\"client_email\"], \"json_keys\", prt\n",
        "            )\n",
        "            print(f\"There are {len(json_keys)} json keys in {sa_path}\")\n",
        "        elif json_keys:\n",
        "            return [i[\"client_email\"] for i in json_keys]\n",
        "        # else:\n",
        "        #     print(f\"No json keys in path: {sa_path} matching filter: {filter}\")\n",
        "\n",
        "    keys = jsons\n",
        "\n",
        "    def all(self):\n",
        "        \"\"\"List all drives, projects, service accounts, json keys, groups and group members.\n",
        "        Also exports these lists with full data fields to csv and xlsx files in data_path folder\"\"\"\n",
        "        self.drives()\n",
        "        self.projects()\n",
        "        self.sas()\n",
        "        self.groups()\n",
        "        self.members()\n",
        "        self.jsons()\n",
        "\n",
        "    def count(self):\n",
        "        \"\"\"Count drives, projects, service accounts, json keys, groups and group members.\"\"\"\n",
        "        list_drives = []\n",
        "        list_drives = self.drives(\"\", \"\", token, False)\n",
        "        print(\"Drive count  :\", len(list_drives))\n",
        "        list_projects = []\n",
        "        list_projects = self.projects(\"\", \"\", token, False)\n",
        "        print(\"Project count:\", len(list_projects))\n",
        "        list_groups = []\n",
        "        list_groups = self.groups(\"\", \"\", group_token, False)\n",
        "        print(\"Group count  :\", len(list_groups))\n",
        "        try:\n",
        "          list_jsons = self.jsons(sa_path, \"\", \"\", False)\n",
        "          print(\"JSON count   :\", len(list_jsons))\n",
        "        except:\n",
        "          print(\"JSON count   : 0\")\n",
        "        list_sas = []\n",
        "        list_sas, _ = self.sas(\"\", False, \"\", token, False)\n",
        "        print(\"SA count     :\", len(list_sas))\n",
        "        list_members = []\n",
        "        list_members = self.members(\"\", \"\", group_token, False)\n",
        "        print(\"Member count :\", len(list_members))\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2NcemUMdp92X",
        "colab_type": "text"
      },
      "source": [
        "###### Add"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jL9KMGLDp8X5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class Add(Help):\n",
        "    \"\"\"Add projects, drives, service accounts(SAs), SA keys/jsons and group members\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Add, self).__init__()\n",
        "        self._list = List()\n",
        "\n",
        "    def projects(\n",
        "        self,\n",
        "        num_new_projects=num_new_projects,\n",
        "        next_project_num=next_project_num,\n",
        "        project_prefix=project_prefix,\n",
        "        retry=5,\n",
        "        token=token,\n",
        "        ppad=4,\n",
        "        prt=False,\n",
        "    ):\n",
        "        \"\"\"Create projects in authorized account. Usage: 'safire add projects 1'.\n",
        "        Uses defaults in config if none specified.\"\"\"\n",
        "        cloud = self._svc(*CLOUD, token)\n",
        "        batch = BatchJob(cloud)\n",
        "        start_projs = self._list.projects(\"\", \"\", token, prt)\n",
        "        start_proj_count = len(start_projs)\n",
        "        num_proj_created = 0\n",
        "        curr_projects = []\n",
        "        proj_count = 0\n",
        "        print(f\"Creating {num_new_projects} new projects\")\n",
        "        retries = retry\n",
        "        while num_new_projects and retries:\n",
        "            new_projs = [\n",
        "                self._pre_pad(project_prefix, ppad, next_project_num + i)\n",
        "                for i in range(num_new_projects)\n",
        "            ]\n",
        "            for proj in new_projs:\n",
        "                batch.add(\n",
        "                    cloud.projects().create(body={\"project_id\": proj, \"name\": proj})\n",
        "                )\n",
        "            batch.execute()\n",
        "            retries -= 1\n",
        "            curr_projects = self._list.projects(\"\", \"\", token, prt)\n",
        "            proj_count = len(curr_projects)\n",
        "            num_proj_created = proj_count - start_proj_count\n",
        "            next_project_num = next_project_num + num_proj_created\n",
        "            num_new_projects = num_new_projects - num_proj_created\n",
        "        if num_proj_created < 1:\n",
        "            print(\n",
        "                \"0 projects created. Your project prefix + number may be used already\",\n",
        "                \"somewhere else in Google. Try changing your prefix.\",\n",
        "            )\n",
        "            exit()\n",
        "        new_projs = [i for i in curr_projects if i not in start_projs]\n",
        "        print(f\"\\nCreated {num_proj_created} projects:\", *new_projs, sep=\"\\n\")\n",
        "        print(f\"Total project count = {proj_count}\")\n",
        "        print(\n",
        "            \"Sleep briefly to allow backend to register projects. Then enabling services.\\n\"\n",
        "        )\n",
        "        sleep(sleep_time)\n",
        "        for proj in new_projs:\n",
        "            self.apis(proj)\n",
        "\n",
        "    def apis(\n",
        "        self, filter=\"\", svcs_to_enable=svcs_to_enable, token=token, prt=False\n",
        "    ):\n",
        "        \"\"\"Enables apis for projects. 'drive' and 'iam' apis by default. Automatic when projects are\n",
        "        created but can be run manually also.\"\"\"\n",
        "        svcusage = self._svc(*SVCUSAGE, token)\n",
        "        projId_list = self._list.projects(filter, \"\", token, prt)\n",
        "        batch = BatchJob(svcusage)\n",
        "        svcs_to_enable = [i + \".googleapis.com\" for i in svcs_to_enable]\n",
        "        for project in projId_list:\n",
        "            for svc1 in svcs_to_enable:\n",
        "                print(f\"Enabling service: {svc1} in project: {project}\")\n",
        "                batch.add(\n",
        "                    svcusage.services().enable(\n",
        "                        name=f\"projects/{project}/services/{svc1}\"\n",
        "                    )\n",
        "                )\n",
        "            batch.execute()\n",
        "\n",
        "    def sas(\n",
        "        self,\n",
        "        filter=\"\",\n",
        "        sas_per_project=sas_per_project,\n",
        "        email_prefix=email_prefix,\n",
        "        next_sa_num=next_sa_num,\n",
        "        retry=retry,\n",
        "        exact=False,\n",
        "        file_tag=\"\",\n",
        "        prt=False,\n",
        "    ):\n",
        "        \"\"\"Create N service accounts/SAs in projects which match 'filter'. Usage: 'safire add sas xyz 5'\n",
        "         will add 5 SAs to all projects containing 'xys' if fewer than 100 exist. Will not overwrite SAs.\"\"\"\n",
        "        iam = self._svc(*IAM, token)\n",
        "        projId_list = self._list.projects(filter, file_tag, token, prt)\n",
        "        all_sas = []\n",
        "        for project in projId_list:\n",
        "            # batch = BatchJob(iam)\n",
        "            sa_emails, _ = self._list.sas(project, True, file_tag, token, False)\n",
        "            start_sa_emails = sa_emails\n",
        "            start_sa_count = len(sa_emails)\n",
        "            all_sas = all_sas + sa_emails\n",
        "            count = min(sas_per_project, 100 - len(sa_emails))\n",
        "            print(\n",
        "                len(sa_emails), \"SAs exist. Creating\", count, \"SAs in project\", project\n",
        "            )\n",
        "            new_sas = []\n",
        "            retries = retry\n",
        "            while count and retries:\n",
        "                batch = BatchJob(iam)\n",
        "                for _ in range(count):\n",
        "                    while [s for s in all_sas if str(next_sa_num) in s.split(\"@\")[0]]:\n",
        "                        next_sa_num += 1\n",
        "                    sa_id = self._pre_pad(email_prefix, spad, next_sa_num)\n",
        "                    new_sas.append(sa_id)\n",
        "                    next_sa_num += 1\n",
        "                    name = f\"projects/{project}\"\n",
        "                    body = {\n",
        "                        \"accountId\": sa_id,\n",
        "                        \"serviceAccount\": {\"displayName\": sa_id},\n",
        "                    }\n",
        "                    batch.add(\n",
        "                        iam.projects().serviceAccounts().create(name=name, body=body)\n",
        "                    )\n",
        "                batch.execute()\n",
        "                retries -= 1\n",
        "                sa_emails, _ = self._list.sas(project, True, file_tag, token, False)\n",
        "                curr_sa_count = len(sa_emails)\n",
        "                count = count - curr_sa_count + start_sa_count\n",
        "                sleep(sleep_time/5)\n",
        "            new_sa_emails = [i for i in sa_emails if i not in start_sa_emails]\n",
        "            num_sas_created = len(new_sa_emails)\n",
        "            print(\n",
        "                f\"\\nCreated {num_sas_created} sas in {project}:\",\n",
        "                *new_sa_emails,\n",
        "                sep=\"\\n\",\n",
        "            )\n",
        "            print(f\"Total SAs in {project} count = {len(sa_emails)}\")\n",
        "\n",
        "    def jsons(\n",
        "        self,\n",
        "        filter=\"\",\n",
        "        sa_path=sa_path,\n",
        "        next_json_num=next_json_num,\n",
        "        json_prefix=json_prefix,\n",
        "        jpad=jpad,\n",
        "        retry=retry,\n",
        "        file_tag=\"\",\n",
        "        token=token,\n",
        "        prt=False,\n",
        "    ):\n",
        "        \"\"\"Create and download json/key files to svcaccts folder. Add to TDs and/or groups.\"\"\"\n",
        "        iam = self._svc(*IAM, token)\n",
        "        projId_list = self._list.projects(filter, file_tag, token, prt)\n",
        "        for project in sorted(projId_list):\n",
        "            batch = BatchJob(iam)\n",
        "            _, sa_uniqueId = self._list.sas(project, True, file_tag, token, prt)\n",
        "            num_sas = len(sa_uniqueId)\n",
        "            print(f\"Downloading {str(len(sa_uniqueId))} SA keys in project {project}\")\n",
        "            resp = []\n",
        "            retries = retry\n",
        "            while len(resp) < num_sas and retries:\n",
        "                for sa in sa_uniqueId:\n",
        "                    batch.add(\n",
        "                        iam.projects()\n",
        "                        .serviceAccounts()\n",
        "                        .keys()\n",
        "                        .create(\n",
        "                            name=f\"projects/{project}/serviceAccounts/{sa}\",\n",
        "                            body={\n",
        "                                \"privateKeyType\": \"TYPE_GOOGLE_CREDENTIALS_FILE\",\n",
        "                                \"keyAlgorithm\": \"KEY_ALG_RSA_2048\",\n",
        "                            },\n",
        "                        )\n",
        "                    )\n",
        "                resp = [i[\"response\"] for i in batch.execute()]\n",
        "                retries -= 1\n",
        "            for i in resp:\n",
        "                if i is not None:\n",
        "                    k = (\n",
        "                        i[\"name\"][i[\"name\"].rfind(\"/\") :],\n",
        "                        b64decode(i[\"privateKeyData\"]).decode(\"utf-8\"),\n",
        "                    )[1]\n",
        "                    json_name = self._pre_pad(json_prefix, jpad, next_json_num)\n",
        "                    with open(\"%s/%s.json\" % (sa_path, json_name), \"w+\") as f:\n",
        "                        f.write(k)\n",
        "                    next_json_num += 1\n",
        "            sleep(sleep_time)\n",
        "\n",
        "    keys = jsons\n",
        "\n",
        "    def drive(self, td_name):\n",
        "        \"\"\"Create a team/shared drive. Usage: 'safire add drive some_name'\"\"\"\n",
        "        drive = self._svc(*DRIVE, token)\n",
        "        body = {\"name\": td_name}\n",
        "        driveId = (\n",
        "            drive.drives()\n",
        "            .create(body=body, requestId=str(uuid.uuid4()), fields=\"id,name\")\n",
        "            .execute()\n",
        "            .get(\"id\")\n",
        "        )\n",
        "        print(f\"  Drive ID for {td_name} is {driveId}\")\n",
        "        return td_name, driveId\n",
        "\n",
        "    def drives(self, *td_list):\n",
        "        \"\"\"Create team/shared drives. Usage: 'safire add drives some_filename' containing \n",
        "        a list of drive names OR `safire add drives mytd1 mytd2 mytd3`\"\"\"\n",
        "        td_file = ' '.join(sys.argv[3:]).strip('\\\"')\n",
        "        if os.path.isfile(td_file):\n",
        "            with open(td_file, \"r\") as td_lst:\n",
        "              td_list = [i for i in td_lst]\n",
        "        for td_name in td_list:\n",
        "            print(f\"Creating {td_name}\")\n",
        "            self.drive(td_name.rstrip())\n",
        "\n",
        "    def members(\n",
        "        self,\n",
        "        project_filter,\n",
        "        group_filter,\n",
        "        retry=5,\n",
        "        file_tag=\"\",\n",
        "        group_token=group_token,\n",
        "        prt=False,\n",
        "    ):\n",
        "        \"\"\"'add members' requires two arguments. Both 'project_filter' and 'group_filter' can be either the full\n",
        "        project/group name or a partial name which matches some projects/groups. \n",
        "        You can add SA emails from multiple projects to multiple groups if you wish.\"\"\"\n",
        "        admin = self._svc(*ADMIN, group_token)\n",
        "        projId_list = self._list.projects(project_filter, file_tag, token, prt)\n",
        "        group_list = self._list.groups(group_filter, file_tag, group_token, prt)\n",
        "        for group in group_list:\n",
        "            for project in projId_list:\n",
        "                batch = BatchJob(admin)\n",
        "                sa_emails, _ = self._list.sas(project, True, file_tag, token, False)\n",
        "                if len(sa_emails) == 0:\n",
        "                    print(\"No service accounts found in\", project)\n",
        "                    continue\n",
        "                group_members = self._list.members(group, file_tag, group_token, prt)\n",
        "                add_sas = [i for i in sa_emails if i not in group_members]\n",
        "                if len(add_sas) == 0:\n",
        "                    continue\n",
        "                print(\n",
        "                    f\"Adding {str(len(add_sas))} SAs to group: {group} from project: {project}\"\n",
        "                )\n",
        "                retries = retry\n",
        "                while add_sas and retries:\n",
        "                    retries -= 1\n",
        "                    for email in add_sas:\n",
        "                        batch.add(\n",
        "                            admin.members().insert(\n",
        "                                groupKey=group, body={\"email\": email, \"role\": \"MEMBER\"}\n",
        "                            )\n",
        "                        )\n",
        "                    batch.execute()\n",
        "                    sleep(sleep_time)\n",
        "                    group_members = self._list.members(\n",
        "                        group_filter, file_tag, group_token, prt\n",
        "                    )\n",
        "                    add_sas = [i for i in sa_emails if i not in group_members]\n",
        "                print(\n",
        "                    f\"{len(sa_emails)-len(add_sas)} SA emails from {project} are in {group}. {len(add_sas)} failed.\"\n",
        "                )\n",
        "\n",
        "    def user(self, user, *td_id, role=\"organizer\"):\n",
        "        \"\"\"Add user (typically group name) to shared/team drive(s). Usage: 'safire add mygroup@domain.com td_filter'\"\"\"\n",
        "        for td_filt in td_id:\n",
        "            drives = self._list.drives(td_filt, \"\", token, False)\n",
        "            for td in drives:\n",
        "                if td_filt in td['name']:\n",
        "                    print(f\"Adding {user} to {td['name']} {td['id']}\")\n",
        "                    drive = self._svc(*DRIVE, token)\n",
        "                    body = {\"type\": \"user\", \"role\": role, \"emailAddress\": user}\n",
        "                    drive.permissions().create(body=body, fileId=td['id'], supportsAllDrives=True, fields=\"id\").execute()\n",
        "                else:\n",
        "                    pass\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0882m2pipfVU",
        "colab_type": "text"
      },
      "source": [
        "###### Remove"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzNDhRCHpea1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class Remove(Help):\n",
        "    \"\"\"Delete sas, jsons/keys, drives and group members. Note: 'remove' and 'delete' are equivalent commands\"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Remove, self).__init__()\n",
        "        self._list = List()\n",
        "\n",
        "    def sas(\n",
        "        self,\n",
        "        project_filter,\n",
        "        token=token,\n",
        "        exact=False,\n",
        "        file_tag=\"\",\n",
        "        prt=False,\n",
        "        retry=retry,\n",
        "    ):\n",
        "        \"\"\"Usage: 'safire remove sas filter' where filter is a string to match the projects from which \n",
        "        you want to delete service accounts. To remove all SAs for all projects use \"\" as your filter\"\"\"\n",
        "        projId_list = self._list.projects(project_filter, file_tag, token, prt)\n",
        "        iam = self._svc(*IAM, token)\n",
        "        for project in projId_list:\n",
        "            sas, _ = self._list.sas(project, True, file_tag, token, False)\n",
        "            retries = retry\n",
        "            while sas and retries:\n",
        "                retries -= 1\n",
        "                if len(sas) == 0:\n",
        "                    print(f\"0 service accounts in {project}. Moving to next project\")\n",
        "                    continue\n",
        "                batch = BatchJob(iam)\n",
        "                print(f\"Removing {len(sas)} service accounts from {project}\")\n",
        "                for i in sas:\n",
        "                    name = f\"projects/{project}/serviceAccounts/{i}\"\n",
        "                    batch.add(iam.projects().serviceAccounts().delete(name=name))\n",
        "                batch.execute()\n",
        "                sleep(sleep_time)\n",
        "                sas, _ = self._list.sas(project, True, file_tag, token, False)\n",
        "\n",
        "    def jsons(self, filter=\"\", sa_path=sa_path):\n",
        "        \"\"\"Remove json keys from svcaccts path\"\"\"\n",
        "        _, _, files = next(os.walk(sa_path))\n",
        "        if not files:\n",
        "            return f\"No json keys found in {sa_path}, Nothing to delete\"\n",
        "        delete_files = [i for i in sorted(files) if \".json\" and str(filter) in i]\n",
        "        # print(f\"Files to be deleted:\\n\", *delete_files, sep=\"\\n\")\n",
        "        # yes_no = input(\"Confirm you want to delete all of these files. y/n: \")\n",
        "        # if yes_no.lower() in [\"y\", \"yes\"]:\n",
        "        for file in delete_files:\n",
        "            print(f\"Removing {file}\")\n",
        "            os.remove(f\"{sa_path}/{file}\")\n",
        "        # else:\n",
        "        #     print(\"Deletion of json files aborted\")\n",
        "\n",
        "    def members(\n",
        "        self,\n",
        "        group_filter,\n",
        "        retry=retry,\n",
        "        batch_size=50,\n",
        "        file_tag=\"\",\n",
        "        group_token=group_token,\n",
        "        prt=False,\n",
        "    ):\n",
        "        \"\"\"Remove members from groups. Match 'filter'\"\"\"\n",
        "        admin = self._svc(*ADMIN, group_token)\n",
        "        batch = BatchJob(admin)\n",
        "        group_list = self._list.groups(group_filter, file_tag, group_token, prt)\n",
        "        group_members = []\n",
        "        for group in group_list:\n",
        "            group_members = self._list.members(group, file_tag, group_token, prt)\n",
        "            retries = retry^2\n",
        "            while len(group_members) > 1 and retries:\n",
        "                print(\n",
        "                    f\"Removing {str(len(group_members))} SAs from group: {group}. Batch = {batch_size}\"\n",
        "                )\n",
        "                retries -= 1\n",
        "                for member in group_members[:batch_size]:\n",
        "                    batch.add(admin.members().delete(groupKey=group, memberKey=member))\n",
        "                batch.execute()\n",
        "                batch = BatchJob(admin)\n",
        "                sleep(sleep_time/4)\n",
        "                group_members = self._list.members(group, file_tag, group_token, prt)\n",
        "            print(\n",
        "                f\"{len(group_members)} members remaining in {group} (excluding OWNER)\"\n",
        "            )\n",
        "\n",
        "    def drive(self, teamDriveId, token=token):\n",
        "        \"\"\"Delete a team/shared drive. Usage: 'safire add teamdrive unique ID'. USE CAREFULLY!\n",
        "        Does not work with non-empty drives.\"\"\"\n",
        "        drvsvc = self._svc(*DRIVE, token)\n",
        "        drives = self._list.drives(teamDriveId, \"\", token, False)\n",
        "        print(\"Drives to be removed:\", *drives, sep=\"\\n\")\n",
        "        yes_no = input(\"Confirm you want to delete all of these drives. y/n: \")\n",
        "        if not yes_no.lower() in [\"y\", \"yes\"]:\n",
        "            return \"Deletion of drives aborted\"\n",
        "        for drive in drives:\n",
        "            print(f\"Removing drive: {drive['name']} with id: {drive['id']}\")\n",
        "            drvsvc.drives().delete(driveId=str(drive[\"id\"])).execute()\n",
        "\n",
        "    def drives(self, input_file, token=token):\n",
        "        \"\"\"Delete team/shared drives. Usage: 'safire add teamdrive some_filename' with unique IDs. USE CAREFULLY\"\"\"\n",
        "        td_list = open(input_file, \"r\")\n",
        "        for teamDriveId in td_list:\n",
        "            print(f\"Deleting {teamDriveId}\")\n",
        "            self.drive(teamDriveId.rstrip())\n",
        "\n",
        "    def user(self, user, *td_id, role=\"organizer\", token=token):\n",
        "        \"\"\"Remove user (typically group name) from shared/team drive(s). Usage: 'safire remove mygroup@domain.com td_filter'\"\"\"\n",
        "        drives = self._list.drives(\"\", \"\", token, False)\n",
        "        for td_filt in td_id:\n",
        "            for td in drives:\n",
        "                if td_filt in td['name']:\n",
        "                    drvsvc = self._svc(*DRIVE, token)\n",
        "                    user_info = []\n",
        "                    resp = {'nextPageToken':None}\n",
        "                    while 'nextPageToken' in resp:\n",
        "                        resp = drvsvc.permissions().list(fileId=td['id'],pageSize=100,fields='nextPageToken,permissions(id,emailAddress,role)',supportsAllDrives=True,pageToken=resp['nextPageToken']).execute()\n",
        "                        user_info += resp['permissions']\n",
        "                    for i in user_info:\n",
        "                        if user in i['emailAddress']:\n",
        "                            print(f\"Removing {user} with id {i['id']} from {td['name']} {td['id']}\")\n",
        "                            drvsvc.permissions().delete(permissionId=i['id'], fileId=td['id'], supportsAllDrives=True, fields=\"id\").execute()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEwTp1m6pA_-",
        "colab_type": "text"
      },
      "source": [
        "###### Rename"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qeV1rF4ro9d7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "class Rename:\n",
        "    \"\"\"Rename json/key files to their email prefix, email numeric (omit prefix), uniqId or in a sequence.\n",
        "    Usage: 'safire rename jsons email'  [choice email, email_seq, uniq, seq]\n",
        "    Renaming is repeatable. Can always delete and redownload keys if needed.\"\"\"\n",
        "    def jsons(self, rename_type, dir=f\"{sa_path}/\"):\n",
        "        \"\"\"Usage: 'safire rename jsons email'  [choice email, email_seq, uniq, seq]\"\"\"\n",
        "        import json, os\n",
        "        filelist = os.listdir(dir)\n",
        "        print(\"\\nOriginal filenames:\", *sorted(filelist), sep=\"\\n\")\n",
        "        new_name = []\n",
        "        if rename_type == \"seq\":\n",
        "            [\n",
        "                os.rename(dir + file, dir + f\"{i+1}.json\")\n",
        "                for i, file in enumerate(sorted(filelist))\n",
        "            ]\n",
        "        else:\n",
        "            for file in sorted(filelist):\n",
        "                try:\n",
        "                    with open(dir + file) as f:\n",
        "                        data = json.load(f)\n",
        "                        if rename_type == \"email\":\n",
        "                            new_name = data[\"client_email\"].split(\"@\")[0] + \".json\"\n",
        "                        if rename_type == \"email_seq\":\n",
        "                            digits = list(\n",
        "                                filter(\n",
        "                                    lambda x: x.isdigit(),\n",
        "                                    data[\"client_email\"].split(\"@\")[0],\n",
        "                                )\n",
        "                            )\n",
        "                            new_name = \"\".join(digits) + \".json\"\n",
        "                        if rename_type == \"uniq\":\n",
        "                            new_name = data[\"client_id\"] + \".json\"\n",
        "                        if os.path.exists(dir + new_name):\n",
        "                            continue\n",
        "                        os.rename(dir + file, dir + new_name)\n",
        "                except:\n",
        "                    continue\n",
        "        print(\"\\nCurrent filenames:\", *sorted(os.listdir(dir)), sep=\"\\n\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmK4-LXJMcnU",
        "colab_type": "text"
      },
      "source": [
        "Add some command line tests\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzeQ-rQXAe_L",
        "colab_type": "text"
      },
      "source": [
        "## Commands"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yiEEfpW_zeB9",
        "colab_type": "text"
      },
      "source": [
        "- Enter command(s) and select `Runtime, Run All` from the top menu in Colab (or use ctrl-f9 to run all). \n",
        "- See below for results of each command. All results are also exported to xlsx and csv files in the /data directory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tTSI9mtOymet",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "if \"safire_ip.py\" == sys.argv[0] and __name__ == \"__main__\":\n",
        "    fire.Fire(Commands)\n",
        "else:\n",
        "    %%time\n",
        "    command1 = \"\" #@param {type:\"string\"}\n",
        "    if command1:\n",
        "        sys.argv = [\"\"] + list(command1.split(\" \"))\n",
        "        print(f\"safire command: {command1} \\n\\nOutput: \\n\")\n",
        "        fire.Fire(Commands)\n",
        "\n",
        "    command2 = \"\" #@param {type:\"string\"}\n",
        "    if command2:\n",
        "        sys.argv = [\"\"] + list(command2.split(\" \"))\n",
        "        print(f\"safire command: {command2} \\n\\nOutput: \\n\")\n",
        "        fire.Fire(Commands)\n",
        "\n",
        "    command3 = \"\" #@param {type:\"string\"}\n",
        "    if command3:\n",
        "        sys.argv = [\"\"] + list(command3.split(\" \"))\n",
        "        print(f\"safire command: {command3} \\n\\nOutput: \\n\")\n",
        "        fire.Fire(Commands)\n",
        "\n",
        "    command4 = \"\" #@param {type:\"string\"}\n",
        "    if command4:\n",
        "        sys.argv = [\"\"] + list(command4.split(\" \"))\n",
        "        print(f\"safire command: {command4} \\n\\nOutput: \\n\")\n",
        "        fire.Fire(Commands)\n",
        "\n",
        "    command5 = \"\" #@param {type:\"string\"}\n",
        "    if command5:\n",
        "        sys.argv = [\"\"] + list(command5.split(\" \"))\n",
        "        print(f\"safire command: {command5} \\n\\nOutput: \\n\")\n",
        "        fire.Fire(Commands)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eojC1KSyFlf3",
        "colab_type": "text"
      },
      "source": [
        "###### Vars and values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F2vRbufNwJg6",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@markdown\n",
        "%whos"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}