As professionals in the IT sector, we all have manual processes in our day-to-day work that we could evolve to some kind of automation or at least semi-automation so that the next time we have to carry out this process again, it can be re-executed in the shortest possible time.
By moving from initially writing our idea in textual form and then taking it to code, we can achieve a standard of work that allows us to reduce time in the most mechanical tasks.
Before talking about tools, we review the principles of automation.
¿What is automation?
- Process automation is the operationalisation of a process that was previously executed manually, using technology and the integration of systems and data.
- The main objective of process automation is to improve workflow progress in an organisation.
- It is possible to replace manual activities with automated ones by using software and systems to support different activities.
Types of automation
RPA (Robotic Process Automation)
Used to automate repetitive software tasks that are usually handled by people, whether they are simple (e.g. filling in forms and preparing invoices) or complex (e.g. customer service and problem solving, chatbot).
Synthetix de Datadope is an RPA-type process.
Another example of an RPA process
BPA (Business Process Automation)
Consists of using software systems to automate repeatable business operations involving several steps. Unlike RPA, these solutions are often complex as they involve several systems and require in-depth analysis (e.g. approval of loan applications, processing of expense reports or a customer’s request for new monitoring in Datadope).
Example of a very simplified BPA process in Datadope
BPM (Business Process Management)
Unlike BPA, which is a strategy to improve specific processes, BPM is a practice that takes an enterprise-wide approach. It is the first step towards the generalisation of business process automation within an enterprise and dictates the lifecycle management theory of BPA.
- It is the strategy we need to follow in order to start doing BPA within a company.
- There are software packages called BPMs that centralise and simplify such simplify such implementations, with visualisations of flows and are generally of flows and in general they are “low code” solutions.
Benefits of automation
The main benefits of automating process operations that are most often cited are:
COST REDUCTION
Every organization faces global pressure to increase its profitability. One approach to achieving this is cost reduction.
+
PRODUCTIVITY
As an organization’s technology needs grow, productivity becomes a bigger concern.
+
RELIABILITY
Automated processes ensure that jobs are not forgotten or run out of sequence, that the data being entered is correct, and that any special processing is done as necessary.
=
+ PERFORMANCE
Automation software can help solve these challenges and help increase system performance
Jupyter Notebooks
Introduction
Let’s imagine that we can write down in a book what we want or have to do (like a recipe book) and then execute those actions or the ones we are interested in, but it’s not magic!
This tool allows us to write notes in Markdown format, indicating for example what is going to be shown or what is being done and then writing code in Python allows us to perform the necessary actions and display them if necessary.
Using Jupyter notebooks we have centralised documentation, code and output.
- It is the base tool on which the solutions of most Cloud products of the same type (Google Colab, AWS SageMaker, …) are built.
- It is based and built on open standards so it can be easily integrated with other technologies (HTML, CSS, Markdown, …).
- It has no specific purpose so it can be used for a multitude of use cases
- Jupyter is open-source
- It is a collaborative tool, notebooks can be easily shared and anyone authorised can change them
- The notebooks are versionable and you can also integrate the notebooks with git
- Functionality can be extended through extensions
To install the native Jupyter Notebooks application, the following instructions can be followed:
- You can also use the Docker image directly.
- Or you can directly use a Cloud implementation such as Google Colab
Architecture
Notebook
- Notebook documents (Notebooks) are documents containing code and rich text elements (paragraph, equations, figures, links, etc.) are saved in .ipynb (json) format and contain the instructions to be executed by the Notebook server.
Notebook Server
- The Jupyter application is a client-server application that allows editing and running notebook documents through a web browser.
- The Jupyter server can be run on a local desktop that does not require Internet access or it can be installed on a remote server and accessed through another network.
Kernel
- The kernel is a “computational engine” that executes the code contained in a Notebook document. The ipython kernel executes python code. There are kernels for many other languages (official kernels).
- Kernels run at the “runtime” level, i.e. every time we open (instantiate) a notebook a kernel is made available to run. When it shuts down, the kernel is shut down.
- Depending on the type of computation, the kernel can consume a significant amount of CPU and RAM. Note that RAM is not freed until the kernel is shut down.
Known extensions
- nteract: Enhanced notebook editing and testing UI, simply create and edit your notebooks and publish them easily for execution in a Jupyter kernel. You only need a kernel installed to run it.
- papermill: Run your notebooks, so you can parameterise and run the code boxes via command line or with Python.
By using this extension correctly we can chain the execution of several notebooks using a single input point.
Use cases
Being such a versatile tool and not having a specific purpose, it can be used for infinite use cases:
- As documents for scientific experiments
- As a documentation centre with examples
- As part of the data exploitation platform of large companies (see example Netflix)
- As procedures or part of automated or semi-automated RPA or BPA procedures.
Practical example of automation with Jupyter
For this example, Google Colab’s implementation of Jupyter is used because of its ease of implementation.
Open the example notebook with this link.
A common day-to-day case could be that as part of the delivery of projects to customers we have to generate a documentation and furthermore, this documentation always follows the same structure although it differs in content.
Would it be a feasible process to generate such documentation with less effort? Probably. We could go through the steps that a human would follow to generate such documentation:
-
- Create a new document or copy it from an existing one.
- Based on the information we know as a basis, generate the document, reformat, etc.
- Save the document
- Upload the document
Can this process be automated? Totally. It is an RPA type process. If we are going to do it in Jupyter with Python Kernel, we simply have to think and locate libraries that allow us to open, manipulate, render and save documents in a known format such as .odt and also use Jupyter’s own input options.
We could also use for example the secretary library, which allows us to render odt documents with Jinja markup (a very common Python format).
Example of an odt document with jinja markings
- The template is created in odt
- It is uploaded to a directory in Google Drive.
- The Notebook is built
- You can run it on demand.
Try to make the Notebook work by filling in the inputs and executing the code boxes, check that the new rendered document is correct based on our inputs.
Final conclusions
- It may not be the best tool to do BPM as it is not designed for that purpose, but it is open source and its development possibilities are endless. It also allows an organisation of the books and even if some parts cannot be automated, they are documented.
- The important thing is not always the tool but to start somewhere, it can be a good place to start thinking about which day-to-day processes we can document or automate part of them and improve them.
