{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PGM JSON Converter\n",
    "\n",
    "The PGM JSON Converter, converts to and from the native power-grid-model JSON data format, but with the addition of\n",
    "handling _extra info_; information about the components, stored in the json data, but not used for the calculations.\n",
    "This json format makes it convenient to evaluate the grid data in perspective of input to power-grid-model. \n",
    "More details about the JSON format are mentioned in `power_grid_model.utils` and an example of it is in [Make Test\n",
    "Dataset](power-grid-model:examples/Make%20Test%20Dataset.ipynb) in power-grid-model repository.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load the JSON data\n",
    "\n",
    "Define source and destination paths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "source_file = \"data/tiny-net/input.json\"\n",
    "destination_file = \"data/tiny-net/sym_output.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For debugging purposes, let's check the JSON input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<pre style='max-height: 160px; overflow: scroll; white-space: pre'>{\n",
       "  \"node\":\n",
       "    [\n",
       "      {\"id\": 1, \"u_rated\": 10500.0, \"name\": \"First Node\"},\n",
       "      {\"id\": 2, \"u_rated\": 10500.0, \"name\": \"Second Node\"},\n",
       "      {\"id\": 3, \"u_rated\": 10500.0, \"name\": \"Third Node\"}\n",
       "    ],\n",
       "  \"line\":\n",
       "    [\n",
       "      {\"id\": 4, \"from_node\": 1, \"to_node\": 2, \"from_status\": 1, \"to_status\": 1, \"r1\": 0.11, \"x1\": 0.12, \"c1\": 4.1380285203892784e-05, \"tan1\": 0.1076923076923077, \"i_n\": 510.0},\n",
       "      {\"id\": 5, \"from_node\": 2, \"to_node\": 3, \"from_status\": 1, \"to_status\": 1, \"r1\": 0.15, \"x1\": 0.16, \"c1\": 5.411268065124442e-05, \"tan1\": 0.10588235294117646, \"i_n\": 520.0}\n",
       "    ],\n",
       "  \"source\":\n",
       "    [\n",
       "      {\"id\": 6, \"node\": 1, \"status\": 1, \"u_ref\": 1.01047619047619, \"sk\": 200000000.0}\n",
       "    ],\n",
       "  \"sym_load\":\n",
       "    [\n",
       "      {\"id\": 7, \"node\": 2, \"status\": 1, \"type\": 4},\n",
       "      {\"id\": 8, \"node\": 3, \"status\": 1, \"type\": 4}\n",
       "    ],\n",
       "  \"sym_voltage_sensor\":\n",
       "    [\n",
       "      {\"id\": 101, \"measured_object\": 1, \"u_sigma\": 105.0, \"u_measured\": 10751.072595758282, \"u_angle_measured\": -0.013054638926306409},\n",
       "      {\"id\": 201, \"measured_object\": 2, \"u_sigma\": 105.0, \"u_measured\": 10752.698591183394, \"u_angle_measured\": -0.01763734945972652},\n",
       "      {\"id\": 301, \"measured_object\": 3, \"u_sigma\": 100.0, \"u_measured\": 10748.320749959701, \"u_angle_measured\": -0.02018233075947404}\n",
       "    ],\n",
       "  \"sym_power_sensor\":\n",
       "    [\n",
       "      {\"id\": 401, \"measured_object\": 4, \"measured_terminal_type\": 0, \"power_sigma\": 37916.0, \"p_measured\": 2412359.2976399013, \"q_measured\": -3024028.886598367},\n",
       "      {\"id\": 402, \"measured_object\": 4, \"measured_terminal_type\": 1, \"power_sigma\": 1000000000000000.0, \"p_measured\": -2000000.0, \"q_measured\": 1000000.0},\n",
       "      {\"id\": 501, \"measured_object\": 5, \"measured_terminal_type\": 0, \"power_sigma\": 20878.0, \"p_measured\": 1230426.390004009, \"q_measured\": -1742195.1033582848},\n",
       "      {\"id\": 502, \"measured_object\": 5, \"measured_terminal_type\": 1, \"power_sigma\": 10435.0, \"p_measured\": -1019999.9999999485, \"q_measured\": -219999.99999999927},\n",
       "      {\"id\": 601, \"measured_object\": 6, \"measured_terminal_type\": 2, \"power_sigma\": 38009.0, \"p_measured\": 2412359.297639887, \"q_measured\": -3024028.8865982923},\n",
       "      {\"id\": 701, \"measured_object\": 7, \"measured_terminal_type\": 4, \"power_sigma\": 10316.0, \"p_measured\": 1010000.0, \"q_measured\": 210000.0},\n",
       "      {\"id\": 801, \"measured_object\": 8, \"measured_terminal_type\": 4, \"power_sigma\": 10435.0, \"p_measured\": 1020000.0, \"q_measured\": 220000.0}\n",
       "    ]\n",
       "}</pre>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from IPython.display import Markdown, display\n",
    "\n",
    "with Path(source_file).open() as json_file:\n",
    "    display(Markdown(f\"<pre style='max-height: 160px; overflow: scroll; white-space: pre'>{json_file.read()}</pre>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Instantiate the converter, optionally with source and destination file paths.\n",
    "Then use `load_input_data()` to load the data and convert it to power-grid-model data.\n",
    "Additional information is stored in `extra_info`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "from power_grid_model_io.converters import PgmJsonConverter\n",
    "\n",
    "converter = PgmJsonConverter(source_file=source_file, destination_file=destination_file)\n",
    "input_data, extra_info = converter.load_input_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's investigate the data we have converted, for one of the components: `nodes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([(1, 10500.), (2, 10500.), (3, 10500.)],\n",
       "      dtype={'names': [id, u_rated], 'formats': ['<i4', '<f8'], 'offsets': [0, 8], 'itemsize': 16, 'aligned': True})"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>u_rated</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>10500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>10500.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  u_rated\n",
       "0   1  10500.0\n",
       "1   2  10500.0\n",
       "2   3  10500.0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{np.int32(1): {'name': 'First Node'},\n",
       " np.int32(2): {'name': 'Second Node'},\n",
       " np.int32(3): {'name': 'Third Node'}}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from power_grid_model import AttributeType, ComponentType\n",
    "\n",
    "# The node data is stored as a numpy structured array in input_data[ComponentType.node]\n",
    "display(input_data[ComponentType.node])\n",
    "\n",
    "# We can use pandas to display the data in a convenient tabular format\n",
    "display(pd.DataFrame(input_data[ComponentType.node]))\n",
    "\n",
    "# Notice that the node names were not stored in the numpy array, as we don't need them for the calculations\n",
    "display({i: extra_info[i] for i in input_data[ComponentType.node][AttributeType.id]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2. Run the calculation\n",
    "\n",
    "Run powerflow calculation with the `input_data` and show the results for `nodes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.023912</td>\n",
       "      <td>10751.072596</td>\n",
       "      <td>-0.013055</td>\n",
       "      <td>2.412359e+06</td>\n",
       "      <td>-3.024029e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1.024067</td>\n",
       "      <td>10752.698591</td>\n",
       "      <td>-0.017637</td>\n",
       "      <td>-1.010000e+06</td>\n",
       "      <td>-2.100000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1.023650</td>\n",
       "      <td>10748.320750</td>\n",
       "      <td>-0.020182</td>\n",
       "      <td>-1.020000e+06</td>\n",
       "      <td>-2.200000e+05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  1.023912  10751.072596 -0.013055  2.412359e+06 -3.024029e+06\n",
       "1   2          1  1.024067  10752.698591 -0.017637 -1.010000e+06 -2.100000e+05\n",
       "2   3          1  1.023650  10748.320750 -0.020182 -1.020000e+06 -2.200000e+05"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from power_grid_model import PowerGridModel\n",
    "\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_state_estimation()\n",
    "\n",
    "display(pd.DataFrame(output_data[ComponentType.node]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Save the data as a JSON file\n",
    "The data can be stored in a json file, using the same converter instance as we used to load the input data. The file will be saved in the `destination_file` path supplied in the constructor, or a custom data_store (file, api endpint, database, etc) can be provided here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "converter.save(data=output_data, extra_info=extra_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Again, for debugging purposes, let's check the JSON output. Notice that the node names are added to the nodes data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<pre style='max-height: 160px; overflow: scroll; white-space: pre'>{\n",
       "  \"node\":\n",
       "    [\n",
       "      {\"id\": 1, \"energized\": 1, \"u_pu\": 1.0239116757780378, \"u\": 10751.072595669397, \"u_angle\": -0.01305463892705287, \"p\": 2412359.297617528, \"q\": -3024028.8865525783, \"name\": \"First Node\"},\n",
       "      {\"id\": 2, \"energized\": 1, \"u_pu\": 1.0240665324851774, \"u\": 10752.698591094362, \"u_angle\": -0.017637349460495298, \"p\": -1009999.9999918179, \"q\": -209999.99999407632, \"name\": \"Second Node\"},\n",
       "      {\"id\": 3, \"energized\": 1, \"u_pu\": 1.0236495952257734, \"u\": 10748.32074987062, \"u_angle\": -0.020182330760257754, \"p\": -1019999.9999917165, \"q\": -219999.99999405813, \"name\": \"Third Node\"}\n",
       "    ],\n",
       "  \"line\":\n",
       "    [\n",
       "      {\"id\": 4, \"energized\": 1, \"loading\": 0.4073282784816051, \"p_from\": 2412359.297617528, \"q_from\": -3024028.8865525783, \"i_from\": 207.7374220256186, \"s_from\": 3868362.4555496555, \"p_to\": -2240426.3899840107, \"q_to\": 1532195.1033373913, \"i_to\": 145.73743890160415, \"s_to\": 2714246.1648914353},\n",
       "      {\"id\": 5, \"energized\": 1, \"loading\": 0.22023469237849028, \"p_from\": 1230426.389992248, \"q_from\": -1742195.1033316434, \"i_from\": 114.52204003681494, \"s_from\": 2132883.7003601748, \"p_to\": -1019999.9999917165, \"q_to\": -219999.99999405813, \"i_to\": 56.04963781958646, \"s_to\": 1043455.7968502964}\n",
       "    ],\n",
       "  \"source\":\n",
       "    [\n",
       "      {\"id\": 6, \"energized\": 1, \"p\": 2412359.297617528, \"q\": -3024028.8865525783, \"i\": 207.7374220256186, \"s\": 3868362.4555496555, \"pf\": 0.6236125299367161}\n",
       "    ],\n",
       "  \"sym_load\":\n",
       "    [\n",
       "      {\"id\": 7, \"energized\": 1, \"p\": 1009999.9999918179, \"q\": 209999.99999407632, \"i\": 55.39027580876722, \"s\": 1031600.697935487, \"pf\": 0.9790609894052049},\n",
       "      {\"id\": 8, \"energized\": 1, \"p\": 1019999.9999917165, \"q\": 219999.99999405813, \"i\": 56.04963781958646, \"s\": 1043455.7968502964, \"pf\": 0.9775210440831494}\n",
       "    ],\n",
       "  \"sym_power_sensor\":\n",
       "    [\n",
       "      {\"id\": 401, \"energized\": 1, \"p_residual\": 2.2372770303036305e-05, \"q_residual\": -4.5788262070800556e-05},\n",
       "      {\"id\": 402, \"energized\": 1, \"p_residual\": 240426.38998401066, \"q_residual\": -532195.1033373913},\n",
       "      {\"id\": 501, \"energized\": 1, \"p_residual\": 1.1760814544459208e-05, \"q_residual\": -2.6641355788115106e-05},\n",
       "      {\"id\": 502, \"energized\": 1, \"p_residual\": -8.23208168299061e-06, \"q_residual\": -5.9411087161009846e-06},\n",
       "      {\"id\": 601, \"energized\": 1, \"p_residual\": 2.2358559448321103e-05, \"q_residual\": -4.5713655083545746e-05},\n",
       "      {\"id\": 701, \"energized\": 1, \"p_residual\": 8.182121646882479e-06, \"q_residual\": 5.92367821461437e-06},\n",
       "      {\"id\": 801, \"energized\": 1, \"p_residual\": 8.283596031333218e-06, \"q_residual\": 5.9418581166426065e-06}\n",
       "    ],\n",
       "  \"sym_voltage_sensor\":\n",
       "    [\n",
       "      {\"id\": 101, \"energized\": 1, \"u_residual\": 8.888256797234817e-08, \"u_angle_residual\": 7.464601853302e-13},\n",
       "      {\"id\": 201, \"energized\": 1, \"u_residual\": 8.903178194685779e-08, \"u_angle_residual\": 7.687774028486416e-13},\n",
       "      {\"id\": 301, \"energized\": 1, \"u_residual\": 8.908074278224376e-08, \"u_angle_residual\": 7.837168414237539e-13}\n",
       "    ]\n",
       "}\n",
       "</pre>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "from IPython.display import Markdown, display\n",
    "\n",
    "with Path(destination_file).open() as json_file:\n",
    "    display(Markdown(f\"<pre style='max-height: 160px; overflow: scroll; white-space: pre'>{json_file.read()}</pre>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap --no-stderr\n",
    "from power_grid_model import PowerGridModel\n",
    "\n",
    "from power_grid_model_io.converters import PgmJsonConverter\n",
    "\n",
    "source_file = \"data/tiny-net/input.json\"\n",
    "destination_file = \"data/tiny-net/sym_output.json\"\n",
    "\n",
    "converter = PgmJsonConverter(source_file=source_file, destination_file=destination_file)\n",
    "input_data, extra_info = converter.load_input_data()\n",
    "pgm = PowerGridModel(input_data=input_data)\n",
    "output_data = pgm.calculate_state_estimation()\n",
    "converter.save(data=output_data, extra_info=extra_info)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "power-grid-model-io",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}